Develop Reference

how to convert datetime to numeric data type?

I have a dataset as
time MachineId
1530677359000000000 01081081
1530677363000000000 01081081
1530681023000000000 01081090
1530681053000000000 01081090
1530681531000000000 01081090
So my codes goes like:
import pandas as pd
from datetime import datetime
import time
import datetime
import matplotlib.pyplot as plt
import matplotlib.dates as mdate
df= pd.read_csv('acn.csv')`
df['time']=pd.to_datetime(df['time'], unit='ns')` #converting the epoch nanosec time to datetime-format
print(df.head())
Output:
time MachineId
0 2018-07-04 04:09:19 1081081.0
1 2018-07-04 04:09:23 1081081.0
2 2018-07-04 05:10:23 1081090.0
3 2018-07-04 05:10:53 1081090.0
4 2018-07-04 05:18:51 1081090.0
and now I want to change my data of time to numeric to generate a plot between time and machine id
dates = plt.dates.date2num(df['time'])
df.plot(kind='scatter',x='dates',y='MachineId')
plt.show()
which throws a error as :
AttributeError: 'module' object has no attribute 'dates'
How can I change datetime format to numeric so that a plot can be formed ?

You got the following error:
AttributeError: 'module' object has no attribute 'dates'
Your error message is telling you that matplotlib.pyplot.dates (plt.dates) doesn't exist. (The error says that there's a module that you're calling 'dates' but it doesn't exist).
So you need to fix that error before you worry about converting anything. Did you mean to call matplotlib.dates.date2num instead? In your code you have the following:
import matplotlib.dates as mdate
So maybe you meant to call mdate.date2num instead? That should eliminate the AttributeError.
If that doesn't work for you, you could try what is suggested in the link provided by one of the other commenters, to use pandas to_pydatetime. I'm not familiar with it, but in this example page, it is accessed as Series.dt.to_pydatetime()
All of this converting is just necessary because you are trying to use df.plot; maybe you should consider just calling matplotlib directly. For example, could you just use plt.plot_date instead? (here's the link to it). Pandas is excellent, but the plotting interface isn't as mature as the rest of it. As an example (I'm not saying this is the exact problem you are having) but here is a known bug in pandas regarding plotting dates. Here is an older stack overflow thread where someone stubs out a plt.plot_date method for you.

You can directly plot dates as well. For example if you want to have the date on the x-axis you pass the dates in ax.plot(df.time, ids). I think this might the closest thing to what you look for.

Checking if a python variable is a date?

One thing that I'm finding hard with the pandas/numpy combo is dealing with dates. My dataframe time series indices are often DateTimeIndexes containing Timestamps but sometimes seem to be something else (e.g. datetime.Date or numpy.datetime64).
Is there a generic way to check if a particular object is a date, i.e. any of the known date variable types? Or is that a function I should look to create myself?
Thanks!

I use this function to convert a series to a consistent datetime object in pandas / numpy. It works with both scalars and series.
import pandas as pd
x = '2018-12-11'
pd.to_datetime(x) # Timestamp('2018-12-11 00:00:00')

if isinstance(yourVariable,datetime.datetime):
print("it's a date")

I would try converting the string representation of what I suspect to be a datetime into a datetime object, using the parse function from dateutil.parser.
https://chrisalbon.com/python/basics/strings_to_datetime/

How to use matplotlib to plot line charts

I use pandas to read my csv file and turn two columns into arrays as independent/dependent variables respectively.
the data reading, array-turning trans and value assign
Then when I want to use matplotlib.pyplot to plot the line charts out, it turns out that 'numpy.ndarray' objects has no attribute 'find'.
import numpy as np
import matplotlib.pyplot as plt
plt.plot(x,y)

The problem is probably with your dtypes, assuming your data are in df check the df.dtypes. Columns you are trying to plot must be numeric (float, int, bool).
I guess that at least one of the columns you are plotting has object dtype, try to find out why (maybe missing values were read as some sort of string, or everything is just considered string) and convert it to correct type with astype, i.e.
df['float_col'] = df['float_col'].astype(np.float64)
Edit:
If you are trying to plot date use, make sure that dtype is actually a date i.e. datetime64[ns] and use matplotlibs dedicated method plot_date

Using statsmodels.seasonal_decompose() without DatetimeIndex but with Known Frequency

I have a time-series signal I would like to decompose in Python, so I turned to statsmodels.seasonal_decompose(). My data has frequency of 48 (half-hourly). I was getting the same error as this questioner, where the solution was to change from an Int index to a DatetimeIndex. But I don't know the actual dates/times my data is from.
In this github thread, one of the statsmodels contributors says that
"In 0.8, you should be able to specify freq as keyword argument to
override the index."
But this seems not to be the case for me. Here is a minimal code example illustrating my issue:
import statsmodels.api as sm
dta = pd.Series([x%3 for x in range(100)])
decomposed = sm.tsa.seasonal_decompose(dta, freq=3)
AttributeError: 'RangeIndex' object has no attribute 'inferred_freq'
Version info:
import statsmodels
print(statsmodels.__version__)
0.8.0
Is there a way to decompose a time-series in statsmodels with a specified frequency but without a DatetimeIndex?
If not, is there a preferred alternative for doing this in Python? I checked out the Seasonal package, but its github lists 0 downloads/month, one contributor, and last commit 9 months ago, so I'm not sure I want to rely on that for my project.

Thanks to josef-pkt for answering this on github. There is a bug in statsmodels 0.8.0 where it always attempts to calculate an inferred frequency based on a DatetimeIndex, if passed a Pandas object.
The workaround when using Pandas series is to pass their values in a numpy array to seasonal_decompose(). For example:
import statsmodels.api as sm
my_pandas_series = pd.Series([x%3 for x in range(100)])
decomposed = sm.tsa.seasonal_decompose(my_pandas_series.values, freq=3)
(no errors)

What is the maximum timestamp numpy.datetime64 can handle?

I'm trying to convert datetime to numpy.datetime64 but the following case fails:
>>> import numpy as np
>>> from datetime import datetime
>>> np.datetime64(datetime.max)
OSError: Failed to use 'localtime_s' to convert to a local time
I presume that datetime64 can't handle such far-dated timestamps.
So what is the maximum timestamp that datetime64 can handle?

Depends on what the specified unit of your np.datetime64 object is (according to the numpy docs). Since you have given a timestamp with microseconds the allowed range is [290301 BC, 294241 AD].
This answered your question but I think the unspoken other question is why it throws an Exception:
I'm facing the same error (using Windows) and I tried a=np.datetime64(datetime.max) which works. Therefore I suspect the problem is NOT the np.datetime64 span (because creating such a datetime works) but that the __repr__ requires the OS in some way and probably the OS limits it in your case. So check what's the maximum localtime of your OS and for every datetime after that you can still work with the np.datetime64 objects but cannot print them on screen. :-)