I work with Series and DataFrames on the terminal a lot. The default __repr__ for a Series returns a reduced sample, with some head and tail values, but the rest missing.
Is there a builtin way to pretty-print the entire Series / DataFrame? Ideally, it would support proper alignment, perhaps borders between columns, and maybe even color-coding for the different columns.
You can also use the option_context, with one or more options:
with pd.option_context('display.max_rows', None, 'display.max_columns', None): # more options can be specified also
print(df)
This will automatically return the options to their previous values.
If you are working on jupyter-notebook, using display(df) instead of print(df) will use jupyter rich display logic (like so).
No need to hack settings. There is a simple way:
print(df.to_string())
Sure, if this comes up a lot, make a function like this one. You can even configure it to load every time you start IPython: https://ipython.org/ipython-doc/1/config/overview.html
def print_full(x):
pd.set_option('display.max_rows', len(x))
print(x)
pd.reset_option('display.max_rows')
As for coloring, getting too elaborate with colors sounds counterproductive to me, but I agree something like bootstrap's .table-striped would be nice. You could always create an issue to suggest this feature.
After importing pandas, as an alternative to using the context manager, set such options for displaying entire dataframes:
pd.set_option('display.max_columns', None) # or 1000
pd.set_option('display.max_rows', None) # or 1000
pd.set_option('display.max_colwidth', None) # or 199
For full list of useful options, see:
pd.describe_option('display')
Use the tabulate package:
pip install tabulate
And consider the following example usage:
import pandas as pd
from io import StringIO
from tabulate import tabulate
c = """Chromosome Start End
chr1 3 6
chr1 5 7
chr1 8 9"""
df = pd.read_table(StringIO(c), sep="\s+", header=0)
print(tabulate(df, headers='keys', tablefmt='psql'))
+----+--------------+---------+-------+
| | Chromosome | Start | End |
|----+--------------+---------+-------|
| 0 | chr1 | 3 | 6 |
| 1 | chr1 | 5 | 7 |
| 2 | chr1 | 8 | 9 |
+----+--------------+---------+-------+
Using pd.options.display
This answer is a variation of the prior answer by lucidyan. It makes the code more readable by avoiding the use of set_option.
After importing pandas, as an alternative to using the context manager, set such options for displaying large dataframes:
def set_pandas_display_options() -> None:
"""Set pandas display options."""
# Ref: https://stackoverflow.com/a/52432757/
display = pd.options.display
display.max_columns = 1000
display.max_rows = 1000
display.max_colwidth = 199
display.width = 1000
# display.precision = 2 # set as needed
set_pandas_display_options()
After this, you can use either display(df) or just df if using a notebook, otherwise print(df).
Using to_string
Pandas 0.25.3 does have DataFrame.to_string and Series.to_string methods which accept formatting options.
Using to_markdown
If what you need is markdown output, Pandas 1.0.0 has DataFrame.to_markdown and Series.to_markdown methods.
Using to_html
If what you need is HTML output, Pandas 0.25.3 does have a DataFrame.to_html method but not a Series.to_html. Note that a Series can be converted to a DataFrame.
If you are using Ipython Notebook (Jupyter). You can use HTML
from IPython.core.display import HTML
display(HTML(df.to_html()))
Try this
pd.set_option('display.height',1000)
pd.set_option('display.max_rows',500)
pd.set_option('display.max_columns',500)
pd.set_option('display.width',1000)
datascroller was created in part to solve this problem.
pip install datascroller
It loads the dataframe into a terminal view you can "scroll" with your mouse or arrow keys, kind of like an Excel workbook at the terminal that supports querying, highlighting, etc.
import pandas as pd
from datascroller import scroll
# Call `scroll` with a Pandas DataFrame as the sole argument:
my_df = pd.read_csv('<path to your csv>')
scroll(my_df)
Disclosure: I am one of the authors of datascroller
Scripts
Nobody has proposed this simple plain-text solution:
from pprint import pprint
pprint(s.to_dict())
which produces results like the following:
{'% Diabetes': 0.06365372374283895,
'% Obesity': 0.06365372374283895,
'% Bachelors': 0.0,
'% Poverty': 0.09548058561425843,
'% Driving Deaths': 1.1775938892425206,
'% Excessive Drinking': 0.06365372374283895}
Jupyter Notebooks
Additionally, when using Jupyter notebooks, this is a great solution.
Note: pd.Series() has no .to_html() so it must be converted to pd.DataFrame()
from IPython.display import display, HTML
display(HTML(s.to_frame().to_html()))
which produces results like the following:
You can set expand_frame_repr to False:
display.expand_frame_repr : boolean
Whether to print out the full DataFrame repr for wide DataFrames
across multiple lines, max_columns is still respected, but the output
will wrap-around across multiple “pages” if its width exceeds
display.width.
[default: True]
pd.set_option('expand_frame_repr', False)
For more details read How to Pretty-Print Pandas DataFrames and Series
this link can help ypu
hi my friend just run this
pd.set_option("display.max_rows", None, "display.max_columns", None)
print(df)
just do this
Output
Column
0 row 0
1 row 1
2 row 2
3 row 3
4 row 4
5 row 5
6 row 6
7 row 7
8 row 8
9 row 9
10 row 10
11 row 11
12 row 12
13 row 13
14 row 14
15 row 15
16 row 16
17 row 17
18 row 18
19 row 19
20 row 20
21 row 21
22 row 22
23 row 23
24 row 24
25 row 25
26 row 26
27 row 27
28 row 28
29 row 29
30 row 30
31 row 31
32 row 32
33 row 33
34 row 34
35 row 35
36 row 36
37 row 37
38 row 38
39 row 39
40 row 40
41 row 41
42 row 42
43 row 43
44 row 44
45 row 45
46 row 46
47 row 47
48 row 48
49 row 49
50 row 50
51 row 51
52 row 52
53 row 53
54 row 54
55 row 55
56 row 56
57 row 57
58 row 58
59 row 59
60 row 60
61 row 61
62 row 62
63 row 63
64 row 64
65 row 65
66 row 66
67 row 67
68 row 68
69 row 69
You can achieve this using below method. just pass the total no. of columns present in the DataFrame as arg to
'display.max_columns'
For eg :
df= DataFrame(..)
with pd.option_context('display.max_rows', None, 'display.max_columns', df.shape[1]):
print(df)
Try using display() function. This would automatically use Horizontal and vertical scroll bars and with this you can display different datasets easily instead of using print().
display(dataframe)
display() supports proper alignment also.
However if you want to make the dataset more beautiful you can check pd.option_context(). It has lot of options to clearly show the dataframe.
Note - I am using Jupyter Notebooks.
I want to add dataframe to excel every time the code executes, in the last row available in the sheet. Here is the code I am using:
import pandas as pd
import pandas
from openpyxl import load_workbook
def append_df_to_excel(df, excel_path):
df_excel = pd.read_excel(excel_path)
result = pd.concat([df_excel, df], ignore_index=True)
result.to_excel(excel_path)
data_set1 = {
'Name': ['Rohit', 'Mohit'],
'Roll no': ['01', '02'],
'maths': ['93', '63']}
df1 = pd.DataFrame(data_set1)
append_df_to_excel(df1, r'C:\Users\kashk\OneDrive\Documents\ScreenStocks.xlsx')
My desired output(after 3 code runs):
Rohit 1 93
Mohit 2 63
Rohit 1 93
Mohit 2 63
Rohit 1 93
Mohit 2 63
But what I get:
Unnamed: 0.1 Unnamed: 0 Name Roll no maths
0 0 0 Rohit 1 93
1 1 1 Mohit 2 63
2 2 Rohit 1 93
3 3 Mohit 2 63
4 Rohit 1 93
5 Mohit 2 63
Not sure where I am going wrong.
It's happening because in a default situation these functions like to_excel or to_csv (and etc.) add a new column with index. So every time you save the file, it adds a new column.
That's why you just should change the raw where you save your dataframe to a file.
result.to_excel(excel_path, index=False)
I am working with Pandas data frame for one of my projects.
I have a column name Count having integer values in that column.
I have 720 values for each hour i.e 24 * 30 days.
I want to run a loop which can get initially first 24 values from the data frame and put in a new column and then take the next 24 and put in the new column and then so on.
for example:
input:
34
45
76
87
98
34
output:
34 87
45 98
76 34
here it is a row of 6 and I am taking first 3 values and putting them in the first column and the next 3 in the 2nd one.
Can someone please help with writing a code/program for the same. It would be of great help.
Thanks!
You can also try numpy's reshape method performed on pd.Series.values.
s = pd.Series(np.arange(720))
df = pd.DataFrame(s.values.reshape((30,24)).T)
Or split (specify how many arrays you want to split into),
df = pd.DataFrame({"day" + str(i): v for i, v in enumerate(np.split(s.values, 30))})
this is a small part of my dataset which contains thousands of rows
designation names runs wickets catches
batsman brendon mccullum 78 0 12
bowler shane bond 0 3 0
bowler mitchell mcclenaghan 20 1 1
batsman kane williamson 192 0 7
wicketkeeper brendon mccullum 78 0 12
batsman daniel vettori 65 11 3
wicketkeeper luke ronchi 7 0 4
bowler daniel vettori 65 11 3
batsman martin guptill 120 0 2
I need to split the dataset based on names, calculate the weightage for each column and then append to same excel sheet. this is my code
df1 = df.sort_values('names')
for i, g in df1.groupby('names'):
g.to_csv('{}'.format(i) + '-names'+ '.csv', header=True, index_label=True)
This code splits the main file into intermediate files for each name and then I run a for loop to perform the calculation on all intermediate files.
filenames = glob.glob('*-names.csv')
for files_ in filenames:
df2 = pd.read_csv(files_)
### perform required calculations
df.to_excel(writer, 'Sheet1', index=False, header=True)
writer.save()
This code is working for me but it creates huge number of intermediate files. I was wondering if there is any method that bypasses that file creation step?
It seems you need processes each group and then write to excel:
df1 = df.sort_values('names')
for i, g in df1.groupby('names'):
print (g)
# perform required calculations with g
g.to_excel(writer, 'Sheet1', index=False, header=True)
writer.save()
Or maybe need apply for each group custom function:
def f(x):
print (x)
# perform required calculations with x
return x
df2 = df1.groupby('names').apply(f)
I have an excel file containing multiple worksheets. Each worksheet contains price & inventory data for individual item codes for a particular month.
for example...
sheetname = 201509
code price inventory
5001 5 92
5002 7 50
5003 6 65
sheetname = 201508
code price inventory
5001 8 60
5002 10 51
5003 6 61
Using pandas dataframe, how is the best way to import this data, organized by time and item code.
I need this dataframe to eventually be able to graph changes in price&inventory for item code 5001 for example.
I would appreciate your help. I am still new to python/pandas.
Thanks.
My solution...
Here is a solution I found to my problem.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
D201509 = pd.read_excel('ExampleSpreadsheet.xlsx', sheetname='201509', index_col='Code')
D201508 = pd.read_excel('ExampleSpreadsheet.xlsx', sheetname='201508', index_col='Code')
D201507 = pd.read_excel('ExampleSpreadsheet.xlsx', sheetname='201507', index_col='Code')
D201506 = pd.read_excel('ExampleSpreadsheet.xlsx', sheetname='201506', index_col='Code')
D201505 = pd.read_excel('ExampleSpreadsheet.xlsx', sheetname='201505', index_col='Code')
total = pd.concat(dict(D201509=D201509, D201508=D201508, D201507=D201507, D201506=D201506, D201505=D201505), axis=1)
total.head()
which will nicely produce this dataframe with hierarchical columns..
Now my new question is how would you plot the change in prices for every code # with this dataframe?
I want to see 5 lines (5001,5002,5003,5004,5005), with the x axis being the time (D201505, D201506, etc) and the y axis being the price value.
Thanks.
This will get your data into a data frame and do a scatter plot on 5001
import pandas as pd
import matplotlib.pyplot as plt
import xlrd
file = r'C:\dickster\data.xlsx'
list_dfs = []
xls = xlrd.open_workbook(file, on_demand=True)
for sheet_name in xls.sheet_names():
df = pd.read_excel(file,sheet_name)
df['time'] = sheet_name
list_dfs.append(df)
dfs = pd.concat(list_dfs,axis=0)
dfs = dfs.sort(['time','code'])
which looks like:
code price inventory time
0 5001 8 60 201508
1 5002 10 51 201508
2 5003 6 61 201508
0 5001 5 92 201509
1 5002 7 50 201509
2 5003 6 65 201509
And now the plot of 5001: price v inventory:
dfs[dfs['code']==5001].plot(x='price',y='inventory',kind='scatter')
plt.show()
which produces: