Develop Reference

Converting a string from long to short based on logic in python

I have multiple long values like the following in a column in pandas dataframe (an example) -
((Type=Food & Value1=Fruit & Value2=Apple) or (Type=Food & Value1=Fruit & Value2=Banana) or (Type=Food & Value1=Vegetable & Value2=Carrot) or (Type=Food & Value1=Vegetable & Value2=Tomato))
I want to convert it to -
((Type=Food & Value1=Fruit & Value2 = Apple|Banana) or (Type=Food & Value1=Vegetable & Value2= Carrot|Tomato))
How can I do it? could not find anything that helps this

((Type=Food & Value1=Fruit & Value2 = Apple|Banana) =>
((Type=Food & Value1=Fruit & ((Value2 = Apple) or (Value2 = Banana))
is this helpful ?

ok I think you need something like this
fruits = ['banana','apple']
print('banana' in fruits)
print('value' in fruits)
output
True
False
for your case:
((Type=Food & Value1=Fruit & Value2 in [Apple,Banana])

Pandas df.to_latex() output gets truncated

Problem: I tried to export a pandas.DataFrame to LaTex using .to_latex().
However, the output for long values (in my case long strings) gets truncated.
Step to reproduce:
import pandas as pd
df = pd.DataFrame(['veryLongString' * i for i in range(1, 5)], dtype='string')
print(df.to_latex())
Output:
\begin{tabular}{ll}
\toprule
{} & 0 \\
\midrule
0 & veryLongString \\
1 & veryLongStringveryLongString \\
2 & veryLongStringveryLongStringveryLongString \\
3 & veryLongStringveryLongStringveryLongStringvery... \\
\bottomrule
\end{tabular}
As you can see, the last row gets truncated (with ...).
I already tried to use the col_space parameter but this does not change the behavior as expected.
It simply shifts the table cells as following:
\begin{tabular}{ll}
\toprule
{} & 0 \\
\midrule
0 & veryLongString \\
1 & veryLongStringveryLongString \\
2 & veryLongStringveryLongStringveryLongString \\
3 & veryLongStringveryLongStringveryLongStringvery... \\
\bottomrule
\end{tabular}
How do I get the full content of the DataFrame exported to Latex?

You can call the context manager with a with statement to temporarily change the max column width:
with pd.option_context("max_colwidth", 1000):
print (df.to_latex())
Output:
\begin{tabular}{ll}
\toprule
{} & 0 \\
\midrule
0 & veryLongString \\
1 & veryLongStringveryLongString \\
2 & veryLongStringveryLongStringveryLongString \\
3 & veryLongStringveryLongStringveryLongStringveryLongString \\
\bottomrule
\end{tabular}
This behaviour is also described here.

After spending some time trying out other parameters from to_latex() as well as other export options, e.g., to_csv(), I was sure that this is not a problem of to_latex().
I found the solution in the pandas documentation:
So the solution is setting this option to None to don't restrict the output (globally).
pd.set_option('display.max_colwidth', None)
Source: https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html

Convert statsmodel table to latex style .png with python

I think I am quite close to solve a problem a lot of statsmodel users had in the past. However, my nonexistent latex experience is slowing down my progress by a lot. :D
Let's jump directly into the problem:
In the following I am using the example dataset from https://www.statsmodels.org/stable/regression.html.
The output is therefore the following regression table:
OLS Regression Results
I have tried several packages to convert the output to a latex style .png. The most promising method seems to be to use the sympy.printing.preview.preview function:
In:
from sympy.printing.preview import preview
preamble = "\\documentclass[12pt]{article}\n" \
"\\usepackage{booktabs,amsmath,amsfonts}\\begin{document}"
preview(res.summary().as_latex(), output='png', filename='output.png', preamble=preamble)
Out:
Latex style .png 1
As you can see, the second and third table of the regression as well as the warning below the table are not in line with the first table.
To fix this, I have tried to use the tabularx package as you can see in the following:
preamble = "\\documentclass[12pt]{article}\n" \
"\\usepackage{booktabs,amsmath,amsfonts, tabularx}\\begin{document}"
preview(res.summary().as_latex().replace("\\begin{tabular}", "\\begin{tabularx}
{\\textwidth}").replace("\\end{tabular}", "\\end{tabularx}"), output='png', filename='output.png', preamble=preamble)
Output:
Latex style .png 2
The horizontal lines of table two and three are aligned with the ones of table one, however the input of the table as well as the warning aren't.
Does anybody know how to continue from here? How can I get the original output table in a latex style saved as a .png with python?
Thank you guys in advance!
Edit1:
Since res.summary().as_latex() outputs the code shown at the end of the post, I have managed to align the tables more or less manually with:
preview(res.summary().as_latex().replace("\\begin{tabular}", "\\begin{tabularx}{20cm}").replace("\\end{tabular}", "\\end{tabularx}").replace("{lclc}", "{>{\hsize=.3\hsize}X >{\hsize=.2\hsize}X >{\hsize=.3\hsize}X >{\hsize=.2\hsize}X}").replace("{lcccccc}", "{XXXXXX p{3.6cm}}"), output='png', filename='output.png', preamble=preamble)
The new Output looks like that. This shouldn't and can't be the final solution, therefore I would like to expand the question if somebody knows a more dynamic (non-manual) way to do this.
Full latex code of the three tables:
\begin{center}
\begin{tabular}{lclc}
\toprule
\textbf{Dep. Variable:} & y & \textbf{ R-squared: } & 0.416 \\
\textbf{Model:} & OLS & \textbf{ Adj. R-squared: } & 0.353 \\
\textbf{Method:} & Least Squares & \textbf{ F-statistic: } & 6.646 \\
\textbf{Date:} & Thu, 04 Feb 2021 & \textbf{ Prob (F-statistic):} & 0.00157 \\
\textbf{Time:} & 18:38:15 & \textbf{ Log-Likelihood: } & -12.978 \\
\textbf{No. Observations:} & 32 & \textbf{ AIC: } & 33.96 \\
\textbf{Df Residuals:} & 28 & \textbf{ BIC: } & 39.82 \\
\textbf{Df Model:} & 3 & \textbf{ } & \\
\bottomrule
\end{tabular}
\begin{tabular}{lcccccc}
& \textbf{coef} & \textbf{std err} & \textbf{t} & \textbf{P$> |$t$|$} & \textbf{[0.025} & \textbf{0.975]} \\
\midrule
\textbf{x1} & 0.4639 & 0.162 & 2.864 & 0.008 & 0.132 & 0.796 \\
\textbf{x2} & 0.0105 & 0.019 & 0.539 & 0.594 & -0.029 & 0.050 \\
\textbf{x3} & 0.3786 & 0.139 & 2.720 & 0.011 & 0.093 & 0.664 \\
\textbf{const} & -1.4980 & 0.524 & -2.859 & 0.008 & -2.571 & -0.425 \\
\bottomrule
\end{tabular}
\begin{tabular}{lclc}
\textbf{Omnibus:} & 0.176 & \textbf{ Durbin-Watson: } & 2.346 \\
\textbf{Prob(Omnibus):} & 0.916 & \textbf{ Jarque-Bera (JB): } & 0.167 \\
\textbf{Skew:} & 0.141 & \textbf{ Prob(JB): } & 0.920 \\
\textbf{Kurtosis:} & 2.786 & \textbf{ Cond. No. } & 176. \\
\bottomrule
\end{tabular}
%\caption{OLS Regression Results}
\end{center}
Notes: \newline
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

regex matching with multiple conditions

I need to obtain a string of letters from the following rows_string variable:
'Equity & 1,638 & \\$3,227,305 & \\$2,649,208 & \\$3,270,402 & \\$3,114,298 & \\$3,173,369 & \\$2,978,769 & \\$3,016,161 & \\$2,807,840\\\\\nFixed Income & 420 & \\$765,856 & \\$661,395 & \\$824,603 & \\$792,579 & \\$794,224 & \\$783,793 & \\$719,307 & \\$630,298\\\\\nCommodities & 119 & \\$72,911 & \\$66,302 & \\$81,649 & \\$81,633 & \\$79,296 & \\$76,450 & \\$64,136 & \\$63,667\\\\\nAsset Allocation & 63 & \\$10,190 & \\$9,275 & \\$10,684 & \\$10,089 & \\$10,371 & \\$9,829 & \\$9,619 & \\$8,880\\\\\nAlternatives & 55 & \\$5,601 & \\$6,023 & \\$6,715 & \\$6,279 & \\$6,365 & \\$6,645 & \\$6,757 & \\$6,243\\\\\nCurrency & 34 & \\$311 & \\$2,014 & \\$1,665 & \\$1,743 & \\$1,683 & \\$1,666 & \\$1,722 & \\$2,058\\\\\nTOTALS & 2,329 & \\$4,082,173 & \\$3,394,217 & \\$4,195,718 & \\$4,006,620 & \\$4,065,308 & \\$3,857,151 & \\$3,817,700 & \\$3,518,986\\\\'
So for instance, I need the following list:
[Equity, Fixed Income, Commodities, Asset Allocation, Alternatives, Currency, Total]
I tried:
re.findall(r'\\\\\n(\w+.*?) &', rows_string)
Great but this omits the "equity" variable
And also gives me an empty list for this string variable
'Starting Portfolio & sell & 21.39\\% & -0.91\\% & 1.52\\% & 9.29\\% & 9.72\\% & 14.89\\% & 38.21\\% & 55.4\\% & & 90.86\\%\\\\'
So for the second string, I need ['Starting Portfolio', 'sell']
What I want is to grab the first item following \\\\\n and first item before the '&' in the string variable. Thank you

You are just missing one \. You are not searching for the letters \ and n but instead for a line break. So just ad \ at the beginning of your regex. Also you are missing the first entry since you state, that your word starts with \\\\\n. To also get the first you could use ^(\w+.*?)|[\\\\\n](\w+.*?) & for example

Try this pattern with re.finditer():
pattern = r"(((?!\\\\\\\\\\n)([a-zA-Z\\s]+))|([a-zA-Z\\s]{2,}\\s?(?!\\&)))"
output_list = [i.group().strip() for i in re.finditer(pattern, rows_string) if i.group().strip()]
Inputs :
s1 = 'Equity & 1,638 & \\$3,227,305 & \\$2,649,208 & \\$3,270,402 & \\$3,114,298 & \\$3,173,369 & \\$2,978,769 & \\$3,016,161 & \\$2,807,840\\\\\nFixed Income & 420 & \\$765,856 & \\$661,395 & \\$824,603 & \\$792,579 & \\$794,224 & \\$783,793 & \\$719,307 & \\$630,298\\\\\nCommodities & 119 & \\$72,911 & \\$66,302 & \\$81,649 & \\$81,633 & \\$79,296 & \\$76,450 & \\$64,136 & \\$63,667\\\\\nAsset Allocation & 63 & \\$10,190 & \\$9,275 & \\$10,684 & \\$10,089 & \\$10,371 & \\$9,829 & \\$9,619 & \\$8,880\\\\\nAlternatives & 55 & \\$5,601 & \\$6,023 & \\$6,715 & \\$6,279 & \\$6,365 & \\$6,645 & \\$6,757 & \\$6,243\\\\\nCurrency & 34 & \\$311 & \\$2,014 & \\$1,665 & \\$1,743 & \\$1,683 & \\$1,666 & \\$1,722 & \\$2,058\\\\\nTOTALS & 2,329 & \\$4,082,173 & \\$3,394,217 & \\$4,195,718 & \\$4,006,620 & \\$4,065,308 & \\$3,857,151 & \\$3,817,700 & \\$3,518,986\\\\'
s2 = 'Starting Portfolio & sell & 21.39\\% & -0.91\\% & 1.52\\% & 9.29\\% & 9.72\\% & 14.89\\% & 38.21\\% & 55.4\\% & & 90.86\\%\\\\'*
Output :
['Equity', 'Fixed Income', 'Commodities', 'Asset Allocation', 'Alternatives', 'Currency', 'TOTALS']
['Starting Portfolio', 'sell']

I don't think there's any reason to focus on the escaped newlines. This should do the trick:
import re
pattern = r'\b[A-Za-z ]*[A-Za-z]\b'
rows_string = 'Equity & 1,638 & \\$3,227,305 & \\$2,649,208 & \\$3,270,402 & \\$3,114,298 & \\$3,173,369 & \\$2,978,769 & \\$3,016,161 & \\$2,807,840\\\\\nFixed Income & 420 & \\$765,856 & \\$661,395 & \\$824,603 & \\$792,579 & \\$794,224 & \\$783,793 & \\$719,307 & \\$630,298\\\\\nCommodities & 119 & \\$72,911 & \\$66,302 & \\$81,649 & \\$81,633 & \\$79,296 & \\$76,450 & \\$64,136 & \\$63,667\\\\\nAsset Allocation & 63 & \\$10,190 & \\$9,275 & \\$10,684 & \\$10,089 & \\$10,371 & \\$9,829 & \\$9,619 & \\$8,880\\\\\nAlternatives & 55 & \\$5,601 & \\$6,023 & \\$6,715 & \\$6,279 & \\$6,365 & \\$6,645 & \\$6,757 & \\$6,243\\\\\nCurrency & 34 & \\$311 & \\$2,014 & \\$1,665 & \\$1,743 & \\$1,683 & \\$1,666 & \\$1,722 & \\$2,058\\\\\nTOTALS & 2,329 & \\$4,082,173 & \\$3,394,217 & \\$4,195,718 & \\$4,006,620 & \\$4,065,308 & \\$3,857,151 & \\$3,817,700 & \\$3,518,986\\\\'
rows = re.findall(pattern, rows_string)
print(rows)
rows_string2 = 'Starting Portfolio & sell & 21.39\\% & -0.91\\% & 1.52\\% & 9.29\\% & 9.72\\% & 14.89\\% & 38.21\\% & 55.4\\% & & 90.86\\%\\\\'
rows2 = re.findall(pattern, rows_string2)
print(rows2)

To get the values, you might use an alternation to either match the words from the start of the string or get the words before &
(?:^[A-Za-z]+(?: [A-Za-z]+)*|[A-Za-z]+(?: [A-Za-z]+)*(?= &))
(?: Non capturing group
^ Start of the line
[A-Za-z]+(?: [A-Za-z]+)* Match 1+ words with only chars A-Za-z
| Or
[A-Za-z]+(?: [A-Za-z]+)*(?= &) Match words followed by &
) Close group
Regex demo | Python demo
For example
import re
pattern = r'(?:^[A-Za-z]+(?: [A-Za-z]+)*|[A-Za-z]+(?: [A-Za-z]+)*(?= &))'
rows_string = 'Equity & 1,638 & \\$3,227,305 & \\$2,649,208 & \\$3,270,402 & \\$3,114,298 & \\$3,173,369 & \\$2,978,769 & \\$3,016,161 & \\$2,807,840\\\\\nFixed Income & 420 & \\$765,856 & \\$661,395 & \\$824,603 & \\$792,579 & \\$794,224 & \\$783,793 & \\$719,307 & \\$630,298\\\\\nCommodities & 119 & \\$72,911 & \\$66,302 & \\$81,649 & \\$81,633 & \\$79,296 & \\$76,450 & \\$64,136 & \\$63,667\\\\\nAsset Allocation & 63 & \\$10,190 & \\$9,275 & \\$10,684 & \\$10,089 & \\$10,371 & \\$9,829 & \\$9,619 & \\$8,880\\\\\nAlternatives & 55 & \\$5,601 & \\$6,023 & \\$6,715 & \\$6,279 & \\$6,365 & \\$6,645 & \\$6,757 & \\$6,243\\\\\nCurrency & 34 & \\$311 & \\$2,014 & \\$1,665 & \\$1,743 & \\$1,683 & \\$1,666 & \\$1,722 & \\$2,058\\\\\nTOTALS & 2,329 & \\$4,082,173 & \\$3,394,217 & \\$4,195,718 & \\$4,006,620 & \\$4,065,308 & \\$3,857,151 & \\$3,817,700 & \\$3,518,986\\\\'
print(re.findall(pattern, rows_string, re.M))
rows_string2 = 'Starting Portfolio & sell & 21.39\\% & -0.91\\% & 1.52\\% & 9.29\\% & 9.72\\% & 14.89\\% & 38.21\\% & 55.4\\% & & 90.86\\%\\\\'
print(re.findall(pattern, rows_string2, re.M))
Output
['Equity', 'Fixed Income', 'Commodities', 'Asset Allocation', 'Alternatives', 'Currency', 'TOTALS']
['Starting Portfolio', 'sell']
If all matches should be followed by & you might simplify the pattern to
[A-Za-z]+(?: [A-Za-z]+)*(?= &)
Regex demo

Assuming your target string (financial keyword) comes after a newline (or start of string) and before & you could do:
>>> re.findall(r'(?:\n|^)([A-Za-z ]+)\s&', s)
['Equity', 'Fixed Income', 'Commodities', 'Asset Allocation', 'Alternatives', 'Currency', 'TOTALS']
This uses some shortcuts, but depending on if you have more complex strings, such as "P&E", "Misc. Expenses", and such, the above might suffice.

Statsmodel summary_col latex format error

I am using the statsmodel's summary_col to give me an output table which summarizes the two regression output.
The code for this is
res3 = summary_col([res1,res2],stars=True,float_format='%0.2f',
info_dict={'R2':lambda x: "{:.2f}".format(x.rsquared)})
f = open('res3.tex', 'w')
f.write(res3.as_latex())
f.close()
I use the res3.tex file as input for another tex file which then generates the results. The problem arises when i convert the table to LaTeX format using as_latex(). The table header shifts to the side in the tex file and looks like this.
The res3.tex file has the following latex code
\begin{table}
\caption{}
\begin{center}
\begin{tabular}{lcc}
\hline
& investment I & investment II \\
\hline
\hline
\end{tabular}
\begin{tabular}{lll}
GDP & 1.35*** & 1.19*** \\
& (0.24) & (0.23) \\
bsent & 0.28*** & 0.26*** \\
& (0.06) & (0.06) \\
rate & -0.22* & -0.65*** \\
& (0.13) & (0.19) \\
research & & 0.80*** \\
& & (0.27) \\
R2 & 0.76 & 0.80 \\
\hline
\end{tabular}
\end{center}\end{table}
The problem seems to arise due to multiple tabular environments. Is there a way to get the investment header on top of the table without manually changing the res3 file (intermediary file)?

You first have to add couple of things to Latex's preamble.
Here you can probably find an answer to your problem Outputting Regressions as Table in Python (similar to outreg in stata)?.
res3 = summary_col([res1,res2],stars=True,float_format='%0.2f',
info_dict={'R2':lambda x: "{:.2f}".format(x.rsquared)})
beginningtex = """\\documentclass{report}
\\usepackage{booktabs}
\\begin{document}"""
endtex = "\end{document}"
f = open('myreg.tex', 'w')
f.write(beginningtex)
f.write(res3.as_latex())
f.write(endtex)
f.close()
All credits to #BKay