Develop Reference

How to replace '\n' in string with '<br>' in Flask app?

I've been trying to modify a string before passing it to my HTML page in Flask (replacing occurrences of '\n' with '<br>'), but the typical methods I use aren't working for some reason.
finalstring = textstring.replace('\n', '<br>')
return render_template('my-form-result.html', emailresponse = finalstring)
This should work, but for some reason, nothing is replaced. How can I get this to work? Thanks!

A better way to replace \n in HTML is using CSS styles.
Your replace() is alright. Debug your code and make sure there is \n before replace().
To be able to view linebreaks in HTML you should use safe filter in the template. But beware that you become open to XSS attacks. To get round this problem you should escape the string before replacing the \n character. This is the code:
from flask import escape
...
...
safe_html = str(escape(text)).replace('\n', '<br/>')
return render_template('[HTML file].html', safe_html=safe_html)
---------
#in the template:
<span> {{ safe_html | safe }} </span>
If you don't use the str() call before replace, then the <br/> will be scaped too. Because the return value from escape() is not string.

Disclaimer: I never worked with Flask, I just looked it up and hope it does what you want to do.
So somewhere in your template my-form-result.html you should find a line containing:
{{ emailresponse }}
You can replace this with:
{% for line in emailresponse.split('\n') %}
{{ line }}
<br />
{% endfor %}
To add an br after every newline

Your replace() code is correct. Make sure you escape the HTML in the template:
{{ emailresponse|safe }}
To diagnose, try this:
finalstring = textstring.replace('\n', '<br>')
print(finalstring)
return render_template('my-form-result.html', emailresponse = finalstring)
Also, show us the source code from the web page, to see what is actually rendering in the template

Django template error while rendering a unicode string

I am passing a python list to a django template like this:
in views.py:
dat = [-77.448599999999999, 37.536900000000003, u'Virginia War Memorial', 1.0]
result={"dat":dat}
return render(request,'result_map.html', {'form': form,'result':result})
and in my template:
var dat = {{ result.dat }}
but in the rendered html I get:
var dat = [-77.448599999999999, 37.536900000000003, u'Virginia War Memorial', 1.0]
which gives an error.
How can I get:
var dat = [-77.448599999999999, 37.536900000000003, 'Virginia War Memorial', 1.0]
thanks

I think you need to pass it through the safe filter if you want the raw string:
{{ result.dat|safe }}
You can also use the {% autoescape off %} block if you want to affect a larger block of the template.

var dat = {{ result.dat }}
You're trying to inject into JavaScript, presumably in a <script> block. You need output that is formatted in JavaScript syntax, but you're implicitly converting a Python list to a string, which will give a Python-syntax list literal, not a JavaScript one. There are a number of differences, including text (Unicode) strings in the list being prefixed with u in Python 2 but not JS.
You're also using the default autoescaping for Django templates, which is HTML-escaping. That would be right for injecting into HTML content, but you're injecting into JavaScript, so you get wrong escapes like ' for '. You can prevent this by adding the |safe filter to avoid the HTML-escaping, but then you have no escaping at all, so your page will be vulnerable to script injection (cross-site scripting).
The right way to output a structured value into a <script> block is:
json.dumps to turn it into a JSON string
replace any instances of the character < in the string with the JSON string literal escape \u003C, so that the string </script> can't be used to escape from the <script> block and inject HTML
mark as safe.
You can combine these operations into a custom filter. Unfortunately Django itself doesn't ship one.
A better approach is to avoid injecting into JavaScript at all. Instead write the json.dumps output into the HTML document using the standard escaping and read it from there using DOM:
return render(request, 'result_map.html', {'form': form, 'dat': json.dumps(dat)})
...
<body data-dat="{{ dat }}">
...
var dat= JSON.parse(document.body.getAttribute('data-dat'));
this allows you to keep your JS completely out of your HTML pages, which would also allow you to use more secure Content-Security-Policy settings.

what's wrong with my python re.sub

this is my code :
string ='''
{% emoji 'MONEY_BAG' %}<span style="color:#7F6C41;">{{ item.name }}を入手した!</span></span>
'''
a = r'''
{%\s+mobile_url\s+['"]{1}(/inventory/view_item/\?)[^'"]*['"]{1}\s+([^%}]+)\s+%}
'''
def aa(x):
print x.group(1)
print x.group(2)
return ''
string = re.sub(a, aa, string)
print string
and it show :
{% emoji 'MONEY_BAG' %}<span style="color:#7F6C41;">{{ item.name }}を入手した!</span></span>
i want to print the x.group(1) and the x.group(2)
so what can i do ,
thanks

It's a bad idea to use regex to extract information from HTML. It's much easier with a HMTL Parser: http://docs.python.org/library/htmlparser.html
Or if you want to crawl a webpage for more information, you might want to use scrapy which is a truly great web crawler framework.

Your extra newline characters in a are causing the regex to never match
a = r'''{%\s+mobile_url\s+['"]{1}(/inventory/view_item/\?)[^'"]*['"]{1}\s+([^%}]+)\s+%}'''

How can i remove <p> </p> with python sub

I have an html file and I want to replace the empty paragraphs with a space.
mystring = "This <p></p><p>is a test</p><p></p><p></p>"
result = mystring.sub("<p></p>" , " ")
This is not working.

Please, don't try to parse HTML with regular expressions. Use a proper parsing module, like htmlparser or BeautifulSoup to achieve this. "Suffer" a short learning curve now and benefit:
Your parsing code will be more robust, handling corner cases you may not have considered that will fail with a regex
For future HTML parsing/munging tasks, you will be empowered to do things faster, so eventually the time investment pays off as well.
You won't be sorry! Profit guaranteed!

I think it's always nice to give an example of how to do this with a real parser, as well as just repeating the sound advice that Eli Bendersky gives in his answer.
Here's an example of how to remove empty <p> elements using lxml. lxml's HTMLParser deals with HTML very well.
from lxml import etree
from StringIO import StringIO
input = '''This <p> </p><p>is a test</p><p></p><p><b>Bye.</b></p>'''
parser = etree.HTMLParser()
tree = etree.parse(StringIO(input), parser)
for p in tree.xpath("//p"):
if len(p):
continue
t = p.text
if not (t and t.strip()):
p.getparent().remove(p)
print etree.tostring(tree.getroot(), pretty_print=True)
... which produces the output:
<html>
<body>
<p>This </p>
<p>is a test</p>
<p>
<b>Bye.</b>
</p>
</body>
</html>
Note that I misread the question when replying to this, and I'm only removing the empty <p> elements, not replacing them with &nbsp. With lxml, I'm not sure of a simple way to do this, so I've created another question to ask:
How can one replace an element with text in lxml?

I think for this particular problem a parsing module would be overkill
simply that function:
>>> mystring = "This <p></p><p>is a test</p><p></p><p></p>"
>>> mystring.replace("<p></p>"," ")
'This <p>is a test</p> '

What if <p> is entered as <P>, or < p >, or has an attribute added, or is given using the empty tag syntax <P/>? Pyparsing's HTML tag support handles all of these variations:
from pyparsing import makeHTMLTags, replaceWith, withAttribute
mystring = 'This <p></p><p>is a test</p><p align="left"></p><P> </p><P/>'
p,pEnd = makeHTMLTags("P")
emptyP = p.copy().setParseAction(withAttribute(empty=True))
null_paragraph = emptyP | p+pEnd
null_paragraph.setParseAction(replaceWith(" "))
print null_paragraph.transformString(mystring)
Prints:
This <p>is a test</p>

using regexp ?
import re
result = re.sub("<p>\s*</p>"," ", mystring, flags=re.MULTILINE)
compile the regexp if you use it often.

I wrote that code:
from lxml import etree
from StringIO import StringIO
html_tags = """<div><ul><li>PID temperature controller</li> <li>Smart and reliable</li> <li>Auto-diagnosing</li> <li>Auto setting</li> <li>Intelligent control</li> <li>2-Rows 4-Digits LED display</li> <li>Widely applied in the display and control of the parameter of temperature, pressure, flow, and liquid level</li> <li> </li> <p> </p></ul> <div> </div></div>"""
document = etree.iterparse(StringIO(html_tags), html=True)
for a, e in document:
if not (e.text and e.text.strip()) and len(e) == 0:
e.getparent().remove(e)
print etree.tostring(document.root)

Convert HTML entities to Unicode and vice versa

How do you convert HTML entities to Unicode and vice versa in Python?

As to the "vice versa" (which I needed myself, leading me to find this question, which didn't help, and subsequently another site which had the answer):
u'some string'.encode('ascii', 'xmlcharrefreplace')
will return a plain string with any non-ascii characters turned into XML (HTML) entities.

You need to have BeautifulSoup.
from BeautifulSoup import BeautifulStoneSoup
import cgi
def HTMLEntitiesToUnicode(text):
"""Converts HTML entities to unicode. For example '&' becomes '&'."""
text = unicode(BeautifulStoneSoup(text, convertEntities=BeautifulStoneSoup.ALL_ENTITIES))
return text
def unicodeToHTMLEntities(text):
"""Converts unicode to HTML entities. For example '&' becomes '&'."""
text = cgi.escape(text).encode('ascii', 'xmlcharrefreplace')
return text
text = "&, ®, <, >, ¢, £, ¥, €, §, ©"
uni = HTMLEntitiesToUnicode(text)
htmlent = unicodeToHTMLEntities(uni)
print uni
print htmlent
# &, ®, <, >, ¢, £, ¥, €, §, ©
# &, ®, <, >, ¢, £, ¥, €, §, ©

Update for Python 2.7 and BeautifulSoup4
Unescape -- Unicode HTML to unicode with htmlparser (Python 2.7 standard lib):
>>> escaped = u'Monsieur le Curé of the «Notre-Dame-de-Grâce» neighborhood'
>>> from HTMLParser import HTMLParser
>>> htmlparser = HTMLParser()
>>> unescaped = htmlparser.unescape(escaped)
>>> unescaped
u'Monsieur le Cur\xe9 of the \xabNotre-Dame-de-Gr\xe2ce\xbb neighborhood'
>>> print unescaped
Monsieur le Curé of the «Notre-Dame-de-Grâce» neighborhood
Unescape -- Unicode HTML to unicode with bs4 (BeautifulSoup4):
>>> html = '''<p>Monsieur le Curé of the «Notre-Dame-de-Grâce» neighborhood</p>'''
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(html)
>>> soup.text
u'Monsieur le Cur\xe9 of the \xabNotre-Dame-de-Gr\xe2ce\xbb neighborhood'
>>> print soup.text
Monsieur le Curé of the «Notre-Dame-de-Grâce» neighborhood
Escape -- Unicode to unicode HTML with bs4 (BeautifulSoup4):
>>> unescaped = u'Monsieur le Curé of the «Notre-Dame-de-Grâce» neighborhood'
>>> from bs4.dammit import EntitySubstitution
>>> escaper = EntitySubstitution()
>>> escaped = escaper.substitute_html(unescaped)
>>> escaped
u'Monsieur le Curé of the «Notre-Dame-de-Grâce» neighborhood'

As hekevintran answer suggests, you may use cgi.escape(s) for encoding stings, but notice that encoding of quote is false by default in that function and it may be a good idea to pass the quote=True keyword argument alongside your string. But even by passing quote=True, the function won't escape single quotes ("'") (Because of these issues the function has been deprecated since version 3.2)
It's been suggested to use html.escape(s) instead of cgi.escape(s). (New in version 3.2)
Also html.unescape(s) has been introduced in version 3.4.
So in python 3.4 you can:
Use html.escape(text).encode('ascii', 'xmlcharrefreplace').decode() to convert special characters to HTML entities.
And html.unescape(text) for converting HTML entities back to plain-text representations.

For python3 use html.unescape():
import html
s = "&"
decoded = html.unescape(s)
# &

$ python3 -c "
> import html
> print(
> html.unescape('&©—')
> )"
&©—
$ python3 -c "
> import html
> print(
> html.escape('&©—')
> )"
&©—
$ python2 -c "
> from HTMLParser import HTMLParser
> print(
> HTMLParser().unescape('&©—')
> )"
&©—
$ python2 -c "
> import cgi
> print(
> cgi.escape('&©—')
> )"
&©—
HTML only strictly requires & (ampersand) and < (left angle bracket / less-than sign) to be escaped. https://html.spec.whatwg.org/multipage/parsing.html#data-state

If someone like me is out there wondering why some entity numbers (codes) like ™ (for trademark symbol), € (for euro symbol) are not encoded properly, the reason is in ISO-8859-1 (aka Windows-1252) those characters are not defined.
Also note that, the default character set as of html5 is utf-8 it was ISO-8859-1 for html4
So, we will have to workaround somehow (find & replace those at first)
Reference (starting point) from Mozilla's documentation
https://developer.mozilla.org/en-US/docs/Web/Guide/Localizations_and_character_encodings

I used the following function to convert unicode ripped from an xls file into a an html file while conserving the special characters found in the xls file:
def html_wr(f, dat):
''' write dat to file f as html
. file is assumed to be opened in binary format
. if dat is nul it is replaced with non breakable space
. non-ascii characters are translated to xml
'''
if not dat:
dat = ' '
try:
f.write(dat.encode('ascii'))
except:
f.write(html.escape(dat).encode('ascii', 'xmlcharrefreplace'))
hope this is useful to somebody

#!/usr/bin/env python3
import fileinput
import html
for line in fileinput.input():
print(html.unescape(line.rstrip('\n')))