With colormath I make a conversion from RGB to xyY value. It works fine for 1 RGB value, but I can't find the right code to do the conversion for multiple RGB values imported from an Excel. I use to following code:
from colormath.color_objects import sRGBColor, xyYColor
from colormath.color_conversions import convert_color
import pandas as pd
data = pd.read_excel(r'C:/Users/User/Desktop/Color/Fontane/RGB/FontaneHuco.xlsx')
df = pd.DataFrame(data, columns=['R', 'G', 'B'])
#print(df)
rgb = sRGBColor(df['R'],df['G'],df['B'], is_upscaled=True)
xyz = convert_color(rgb, xyYColor)
print(xyz)
But when i run this code i receive to following error:
Traceback (most recent call last):
File "C:\Users\User\PycharmProjects\pythonProject4\Overige\Chroma.py", line 9, in <module>
lab = sRGBColor(df['R'], df['G'], df['B'])
File "C:\Users\User\AppData\Local\Programs\Python\Python39\lib\site-packages\colormath\color_objects.py", line 524, in __init__
self.rgb_r = float(rgb_r)
File "C:\Users\User\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\core\series.py", line 141, in wrapper
raise TypeError(f"cannot convert the series to {converter}")
TypeError: cannot convert the series to <class 'float'>
Does anyone has an idea how to fix this problem?
convert_color is expecting floats and you're giving it dataframe columns instead. You need to apply the conversion one row at at time, which can be done as follows:
xyz = df.apply(
lambda row: convert_color(
sRGBColor(row.R, row.G, row.B, is_upscaled=True), xyYColor
),
axis=1,
)
this is my code, I am trying to get business days between two dates
the number of days is saved in a new column 'nd'
import numpy as np
df1 = pd.DataFrame(pd.date_range('2020-01-01',periods=26,freq='D'),columns=['A'])
df2 = pd.DataFrame(pd.date_range('2020-02-01',periods=26,freq='D'),columns=['B'])
df = pd.concat([df1,df2],axis=1)
# Iterate over each row of the DataFrame
for index , row in df.iterrows():
bc = np.busday_count(row['A'],row['B'])
df['nd'] = bc
I am getting this error.
Traceback (most recent call last):
File "<input>", line 35, in <module>
File "<__array_function__ internals>", line 5, in busday_count
TypeError: Iterator operand 0 dtype could not be cast from dtype('<M8[us]') to dtype('<M8[D]') according to the rule 'safe'
Is there a way to fix it or another way to get the solution?
busday_count only accepts dates, not datetimes change
bc = np.busday_count(row['A'],row['B'])
to
np.busday_count(row['A'].date(), row['B'].date())
I need to create a vector of dates with pd.date_range specifying the min and the max for date values.
Date values come from a subselection performed on a dataframe object ds.
This is the code I wrote:
Note that Date in ds are obtained from
ds = pd.read_excel("data.xlsx",sheet_name='all') # Read the Excel file
ds['Date'] = pd.to_datetime(ds['Date'], infer_datetime_format=True)
This is the part inside a for loop where x loops on a list of Names.
for x in lofNames:
date_tmp = ds.loc[ds['Security Name']==x,['Date']]
mindate = date_tmp.min()
maxdate = date_tmp.max()
date = pd.date_range(start=mindate, end=maxdate, freq='D')
This is the error I get:
Traceback (most recent call last):
File "<ipython-input-8-1f56d07b5a74>", line 4, in <module>
date = pd.date_range(start=mindate, end=maxdate, freq='D')
File "/Users/marco/opt/anaconda3/lib/python3.7/site-packages/pandas/core/indexes/datetimes.py", line 1180, in date_range
**kwargs,
File "/Users/marco/opt/anaconda3/lib/python3.7/site-packages/pandas/core/arrays/datetimes.py", line 365, in _generate_range
start = Timestamp(start)
File "pandas/_libs/tslibs/timestamps.pyx", line 418, in pandas._libs.tslibs.timestamps.Timestamp.__new__
File "pandas/_libs/tslibs/conversion.pyx", line 329, in pandas._libs.tslibs.conversion.convert_to_tsobject
TypeError: Cannot convert input [Date 2007-01-09
dtype: datetime64[ns]] of type <class 'pandas.core.series.Series'> to Timestamp
What's wrong?
thank you
Here is returned one column DataFrame instead Series, so next min and max returned one item Series instead scalar, so error is raised:
date_tmp = ds.loc[ds['Security Name']==x,['Date']]
Correct way is removed []:
date_tmp = ds.loc[ds['Security Name']==x,'Date']
So after I converted the UTC timezone in the Time column of my dataframe and saved it to a new csv file, I decided to draw a time plot of frequency of tweets. My time plot was initially working when timezone was UTC but after being converted to Eastern, it gives me the error below. How should I fix it?
import pandas as pd
import matplotlib.pyplot as plt
time_interval = pd.offsets.Second(10)
fig, ax = plt.subplots(figsize=(6, 3.5))
ax = (
pd.read_csv('converted_timezone_tweets.csv', parse_dates=['Time'])
.resample(time_interval, on='Time')['ID']
.count()
.plot.line(ax=ax)
)
plt.show()
And the error is:
/scratch/sjn/anaconda/bin/python /scratch2/debate_tweets/temporal_analysis.py
Traceback (most recent call last):
File "/scratch2/debate_tweets/temporal_analysis.py", line 18, in <module>
pd.read_csv('converted_timezone_tweets.csv', parse_dates=['Time'])
File "/scratch/sjn/anaconda/lib/python3.6/site-packages/pandas/io/parsers.py", line 655, in parser_f
return _read(filepath_or_buffer, kwds)
File "/scratch/sjn/anaconda/lib/python3.6/site-packages/pandas/io/parsers.py", line 411, in _read
data = parser.read(nrows)
File "/scratch/sjn/anaconda/lib/python3.6/site-packages/pandas/io/parsers.py", line 1005, in read
ret = self._engine.read(nrows)
File "/scratch/sjn/anaconda/lib/python3.6/site-packages/pandas/io/parsers.py", line 1748, in read
data = self._reader.read(nrows)
File "pandas/_libs/parsers.pyx", line 890, in pandas._libs.parsers.TextReader.read (pandas/_libs/parsers.c:10862)
File "pandas/_libs/parsers.pyx", line 912, in pandas._libs.parsers.TextReader._read_low_memory (pandas/_libs/parsers.c:11138)
File "pandas/_libs/parsers.pyx", line 966, in pandas._libs.parsers.TextReader._read_rows (pandas/_libs/parsers.c:11884)
File "pandas/_libs/parsers.pyx", line 953, in pandas._libs.parsers.TextReader._tokenize_rows (pandas/_libs/parsers.c:11755)
File "pandas/_libs/parsers.pyx", line 2184, in pandas._libs.parsers.raise_parser_error (pandas/_libs/parsers.c:28765)
pandas.errors.ParserError: Error tokenizing data. C error: Buffer overflow caught - possible malformed input file.
Process finished with exit code 1
converted_timezone_tweets.csv look like this:
,Candidate,ID,Time,Username,Tweet
0,Clinton,788948653016842240,2016-10-19 23:43:11-04:00,Tamayo_castle,Hillary Clinton dresses as Christian Bale at the debate via /r/pics
1,Clinton,788948666501464064,2016-10-19 23:43:14-04:00,ThinkCenter1968,"It's like I told my kids, a reason U don't want 2 vote 4 Hillary is U want the inheritance I'm leaving U, Right? They changed their minds!"
2,Clinton,788948673594097664,2016-10-19 23:43:16-04:00,21stCenRevolt,When hearing about Saudi Arabia murdering people for being gay. Hillary laughed with glee. She disgusting and disgraceful. #debatenight
3,Both,788948662881751040,2016-10-19 23:43:13-04:00,mikeywan,MEGYN IS A PAID HILLARY WHORE #TrumpPence2016 #TrumpTrain
4,Both,788948675313696769,2016-10-19 23:43:16-04:00,erwoti,Can't wait to hear #realDonaldTrump call that Nasty Woman (Hillary Clinton) - Madam President #debatenight #ChrisWallace
5,Clinton,788948671756955650,2016-10-19 23:43:15-04:00,isaac_urner,"The Clinton campaign already has redirecting to their site. That's what a real campaign looks like.
#badhombres2016"
Same code works for valid_tweets.csv and creates a plot like below:
valid_tweets.csv lines look like:
Candidate,ID,Time,Username,Tweet
Clinton,788948653016842240,2016-10-20 03:43:11+00:00,Tamayo_castle,Hillary Clinton dresses as Christian Bale at the debate via /r/pics
Clinton,788948666501464064,2016-10-20 03:43:14+00:00,ThinkCenter1968,"It's like I told my kids, a reason U don't want 2 vote 4 Hillary is U want the inheritance I'm leaving U, Right? They changed their minds!"
Clinton,788948673594097664,2016-10-20 03:43:16+00:00,21stCenRevolt,When hearing about Saudi Arabia murdering people for being gay. Hillary laughed with glee. She disgusting and disgraceful. #debatenight
Both,788948662881751040,2016-10-20 03:43:13+00:00,mikeywan,MEGYN IS A PAID HILLARY WHORE #TrumpPence2016 #TrumpTrain
Both,788948675313696769,2016-10-20 03:43:16+00:00,erwoti,Can't wait to hear #realDonaldTrump call that Nasty Woman (Hillary Clinton) - Madam President #debatenight #ChrisWallace
Clinton,788948671756955650,2016-10-20 03:43:15+00:00,isaac_urner,"The Clinton campaign already has redirecting to their site. That's what a real campaign looks like.
#badhombres2016"
Update:
in my first file I have:
import pandas as pd
import matplotlib.pyplot as plt
#2016-10-20 03:43:11+00:00
tweets_df = pd.read_csv('valid_tweets.csv')
tweets_df['Time'] = pd.Index(pd.to_datetime(tweets_df['Time'], utc=True)).tz_localize('UTC').tz_convert('US/Eastern')
tweets_df.to_csv('converted_timezone_tweets.csv', index=False)
In my second file I have:
import pandas as pd
import matplotlib.pyplot as plt
time_interval = pd.offsets.Second(10)
fig, ax = plt.subplots(figsize=(6, 3.5))
ax = (
pd.read_csv('converted_timezone_tweets.csv', engine='python', parse_dates=['Time'])
.resample(time_interval, on='Time')['ID']
.count()
.plot.line(ax=ax)
)
plt.show()
After using the engine='python' as in one of the answers, I get this error:
/scratch/sjn/anaconda/bin/python /scratch2/debate_tweets/temporal_analysis.py
Traceback (most recent call last):
File "/scratch2/debate_tweets/temporal_analysis.py", line 11, in <module>
.resample(time_interval, on='Time')['ID']
File "/scratch/sjn/anaconda/lib/python3.6/site-packages/pandas/core/generic.py", line 4729, in resample
base=base, key=on, level=level)
File "/scratch/sjn/anaconda/lib/python3.6/site-packages/pandas/core/resample.py", line 969, in resample
return tg._get_resampler(obj, kind=kind)
File "/scratch/sjn/anaconda/lib/python3.6/site-packages/pandas/core/resample.py", line 1091, in _get_resampler
"but got an instance of %r" % type(ax).__name__)
TypeError: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex, but got an instance of 'Index'
Process finished with exit code 1
I did a vimdiff of the first 5 lines of each csv and this is what I get:
It seems like the error is with using the C engine to parse the csv. I'm not knowledgeable enough to know why that might be, but a possible workaround to to force the df.read_csv() bit to use the python engine by passing the engine = 'python' argument. As per the Pandas documentation, pd.read_csv() defaults to using the C engine for speed. Given that your error is hinting at a problem with the C engine, that might be a good place to start. so, try pd.read_csv('converted_timezone_tweets.csv', parse_dates=['Time'], engine = 'python') There was also something on GitHub hinting towards similar problems and fixes
Per the comment, this code
df1 = pd.read_csv('converted_timezone_tweets.csv', engine='python')
mask = pd.isnull(pd.to_datetime(df1['Time'], errors='coerce'))
print(df1.loc[mask, 'Time'])
prints
9941 None
27457 None
27458 None
...
this implies there are a number of entries in converted_timezone_tweets.csv whose Time field is the string 'None'.
You might want to go back and investigate what these values were in your original CSV:
df1 = pd.read_csv('converted_timezone_tweets.csv', engine='python')
mask = pd.isnull(pd.to_datetime(df1['Time'], errors='coerce'))
tweets_df = pd.read_csv('valid_tweets.csv')
print(tweets_df.loc[mask, 'Time'])
If there is no Time data for these tweets perhaps the most sensible thing to do is throw them away since we can't classify what time interval they belong to.
You could use df1 = df1.loc[mask, :] to remove the offending rows:
import pandas as pd
import matplotlib.pyplot as plt
df1 = pd.read_csv('converted_timezone_tweets.csv', engine='python')
df1['Time'] = pd.to_datetime(df1['Time'], errors='coerce')
mask = pd.notnull(df1['Time'])
df1 = df1.loc[mask, :]
df1 = df1.set_index('Time')
counts = df1.resample('10S')['ID'].count()
fig, ax = plt.subplots(figsize=(6, 3.5))
counts.plot.line(ax=ax)
plt.show()
To avoid parsing errors, we call pd.read_csv (above) without setting the parse_dates parameter. So pd.read_csv returns a DataFrame whose Time column contains date strings:
df1 = pd.read_csv('converted_timezone_tweets.csv', engine='python')
# ID Time
# 0 5 2016-10-19 23:43:00-04:00
# 1 5 2016-10-19 23:43:05-04:00
# 2 5 2016-10-19 23:43:10-04:00
# 3 5 2016-10-19 23:43:15-04:00
# ...
We then use pd.to_datetime to parse the date strings into datetimes.
pd.to_datetime parses the date strings by converting them to UTC while taking timezone offsets into account. The resulting datetimes are naive -- no timezone information is attached. This behavior is derived from the underlying NumPy datetime64[ns] data type used by Pandas to represent datetimes.
Therefore, to make the datetimes once again timezone-aware, you would need to call tz_localize/tz_convert again:
df1['Time'] = pd.Index(df1['Time']).tz_localize('UTC').tz_convert('US/Eastern')
But this also shows there was nothing gained by calling tz_convert the first time and storing the result in converted_timezone_tweets.csv the first time.
So a better solution (which does not require calling tz_convert after loading converted_timezone_tweets.csv) is to write converted_timezone_tweets.csv without the timezone offset. You can do that by dropping the timezone offset by calling tz_localize(None):
df1['Time'] = pd.Index(pd.to_datetime(df1['Time'], utc=True)).tz_localize('UTC').tz_convert('US/Eastern').tz_localize(None)
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
N = 10
df = pd.DataFrame({'Time':pd.date_range('2016-10-20 03:43:00', periods=N, freq='5S'), 'ID':np.random.randint(N)})
df1 = df.copy()
df1['Time'] = pd.Index(pd.to_datetime(df1['Time'], utc=True)).tz_localize('UTC').tz_convert('US/Eastern').tz_localize(None)
df1.to_csv('converted_timezone_tweets.csv', index=False)
df1 = pd.read_csv('converted_timezone_tweets.csv', engine='python')
df1['Time'] = pd.to_datetime(df1['Time'], errors='coerce')
mask = pd.notnull(df1['Time'])
df1 = df1.loc[mask, :]
df = df.set_index('Time')
df1 = df1.set_index('Time')
counts1 = df1.resample('10S')['ID'].count()
counts = df.resample('10S')['ID'].count()
fig, ax = plt.subplots(figsize=(6, 3.5), nrows=2)
counts.plot.line(ax=ax[0])
counts1.plot.line(ax=ax[1])
plt.show()
Note that it might be more appealing to store all time-related data in UTC
rather than with respect to some other local timezone. That way, if you have many
CSV files you do not have to keep track of which timezone the time data is
relative to. From this point of view, it would be preferrable to keep
valid_tweets.csv, drop converted_timezone_tweets.csv, and do the conversion to
US/Eastern only when necessary:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv('valid_tweets.csv')
df['Time'] = pd.to_datetime(df['Time'], errors='coerce')
mask = pd.notnull(df['Time'])
df = df.loc[mask, :]
df['Time'] = pd.Index(df['Time']).tz_localize('UTC').tz_convert('US/Eastern')
df = df.set_index('Time')
counts = df.resample('10S')['ID'].count()
fig, ax = plt.subplots(figsize=(6, 3.5))
counts.plot.line(ax=ax)
plt.show()
I am new to python and I am trying to build Time Series through this. I am trying to convert this csv data into time series, however by the internet and stack research, 'result' should have had
<class 'pandas.tseries.index.DatetimeIndex'>,
but my output is not converted time series. Why is it not converting? How do I convert it? Thanks for the help in advance.
import pandas as pd
import numpy as np
import matplotlib.pylab as plt
data = pd.read_csv('somedata.csv')
print data.head()
#selecting specific columns by column name
df1 = data[['a','b']]
#converting the data to time series
dates = pd.date_range('2015-01-01', '2015-12-31', freq='H')
dates #preview
results:
DatetimeIndex(['2015-01-01 00:00:00', '2015-01-01 01:00:00',
...
'2015-12-31 23:00:00', '2015-12-31 00:00:00'],
dtype='datetime64[ns]', length=2161, freq='H')
Above is working, however I get error below:
df1 = Series(df1[:,2], index=dates)
output:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'Series' is not defined
After attempting the pd.Series...
df1 = pd.Series(df1[:,2], index=dates)
Error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/someid/miniconda2/lib/python2.7/site- packages/pandas/core/frame.py", line 1992, in __getitem__
return self._getitem_column(key)
File "/home/someid/miniconda2/lib/python2.7/site- packages/pandas/core/frame.py", line 1999, in _getitem_column
return self._get_item_cache(key)
File "/home/someid/miniconda2/lib/python2.7/site- packages/pandas/core/generic.py", line 1343, in _get_item_cache
res = cache.get(item)
TypeError: unhashable type
you do need to have the pd.Series. However, you were also doing something else wrong. I'm assuming you want to get all rows, 2nd column of df1 and return a pd.Series with an index of dates.
Solution
df1 = pd.Series(df1.iloc[:, 1], index=dates)
Explanation
df1.iloc is used to return the slice of df1 by row/column postitioning
[:, 1] gets all rows, 2nd columns
Also, df1.iloc[:, 1] returns a pd.Series and can be passed into the pd.Series constructor.