Develop Reference

List from configparser passing as argument to telethon errors

I got a little problem that make stuck me at code small script that collect data from telegram chats. For first i will show my code and config file:
Config.ini:
[account]
api_id = xxxx
api_hash = xxxx
[parser]
channels_to_parse = [-xxxx,-xxxx]
Run.py
import configparser
import asyncio
import time
from telethon import events
from telethon import TelegramClient
from telethon.tl import functions, types
from datetime import datetime
#load config file
config = configparser.ConfigParser()
config.read('config.ini', encoding="utf-8")
#telethon init
client = TelegramClient('sess', config.get("account", 'api_id'), config.get("account", 'api_hash'))
client.start()
#main cycle
#client.on(events.NewMessage(chats=config.get('parser' , 'channels_to_parse')))
async def main(event):
#some code...
client.run_until_disconnected()
The main problem goes from string that contains arguments for telethon that points chats IDs from which i collecting data:
ValueError: Cannot find any entity corresponding to "[-xxxx]"
When i passing arguments manually, without configparser:
#client.on(events.NewMessage(chats = [-xxxx, -xxxx]))
Everything works well. So i think that issue related to configparser or configparser parameters. I checked configparser docs and didn't find anything that can help me.
I already tried to use channels name instead IDs. Maybe who's can explain me what i do wrong.

I can't check this at the moment, but I have an idea that configparser returns the str type and you need a list. But I could be wrong(
UPDATE
I checked it out and I was right!
config.ini:
[parser]
channels_to_parse = -xxxx -xxxx
file.py:
parser = config.get('parser', 'channels_to_parse') # type str
chats = [int(i) for i in parser.split()] # type list
#client.on(events.NewMessage(chats=chats))

how to display all snapshots in my aws account using python boto3

the aim of this program is to delete snapshots that are older than 60 days. when run it displays the following error " a=snapshot[s].start_time
AttributeError: 'dict' object has no attribute 'start_time' " This is my code
#!/usr/bin/env python
import boto3
import datetime
client = boto3.client('ec2')
snapshot= client.describe_snapshots()
for s in snapshot:
a=snapshot[s].start_time
b=a.date()
c=datetime.datetime.now().date()
d=c-b
if d.days>60 :
snapshot[s].delete(dry_run=True)

Your error is in the line a=snapshot[s].start_time, use a=s.start_time
Note I would change "snapshot" to "snapshots". then in your for loop:
for snapshot in snapshots:
This makes the code easier to read and clear on what your variables represent.
Another item is that start_time is a string. You will need to parse this to get a number. Here is an example to help you:
delete_time = datetime.utcnow() - timedelta(days=days)
for snapshot in snapshots:
start_time = datetime.strptime(
snapshot.start_time,
'%Y-%m-%dT%H:%M:%S.000Z'
)
if start_time < delete_time:
***delete your snapshot here***

This should do it-
import boto3
import json
import sys
from pprint import pprint
region = 'us-east-1'
ec2 = boto3.client('ec2', region)
resp = ec2.describe_instances()
resp_describe_snapshots = ec2.describe_snapshots(OwnerIds=['*******'])
snapshot = resp_describe_snapshots['Snapshots']
snapshots = [''];
for snapshotIdList in resp_describe_snapshots['Snapshots']:
snapshots.append(snapshotIdList.get('SnapshotId'))
for id in snapshots:
print(id)

Call a function in Python file using Flask

I have python file called testing_file.py:
from datetime import datetime
import MySQLdb
# Open database connection
class DB():
def __init__(self, server, user, password, db_name):
db = MySQLdb.connect(server, user, password, db_name )
self.cur = db.cursor()
def time_statistic(self, start_date, end_date):
time_list = {}
sql = "SELECT activity_log.datetime, activity_log.user_id FROM activity_log"
self.cur.execute(sql)
self.date_data = self.cur.fetchall()
for content in self.date_data:
timestamp = str(content[0])
datetime_object = datetime.strptime(timestamp, '%Y-%m-%d %H:%M:%S')
timestamps = datetime.strftime(datetime_object, "%Y-%m-%d")
if start_dt <= timestamps and timestamps <= end_dt:
if timestamps not in time_list:
time_list[timestamps]=1
else:
time_list[timestamps]+=1
return json.dumps(time_list)
start_date = datetime.strptime(str('2017-4-7'), '%Y-%m-%d')
start_dt = datetime.strftime(start_date, "%Y-%m-%d")
end_date = datetime.strptime(str('2017-5-4'), '%Y-%m-%d')
end_dt = datetime.strftime(end_date, "%Y-%m-%d")
db = DB("host","user_db","pass_db","db_name")
db.time_statistic(start_date, end_date)
I want to access the result (time_list) thru API using Flask. This is what i've wrote so far, doesn't work and also I've tried another way:
from flask import Flask
from testing_api import *
app = Flask(__name__)
#app.route("/")
def get():
db = DB("host","user_db","pass_db","db_name")
d = db.time_statistic()
return d
if __name__ == "__main__":
app.run(debug=True)
Question: This is my first time work with API and Flask. Can anyone please help me thru this. Any hints are appreciated. Thank you
I've got empty list as result {}

There are many things wrong with what you are doing.
1.> def get(self, DB) why self? This function does not belong to a class. It is not an instance function. self is a reference of the class instance when an instance method is called. Here not only it is not needed, it is plain and simple wrong.
2.> If you look into flask's routing declaration a little bit, you will see how you should declare a route with parameter. This is the link. In essence you should something like this
#app.route("/path/<variable>")
def route_func(variable):
return variable
3.> Finally, one more thing I would like to mention, Please do not call a regular python file test_<filename>.py unless you plan to use it as a unit testing file. This is very confusing.
Oh, and you have imported DB from your module already no need to pass it as a parameter to a function. It should be anyway available inside it.

There are quite a few things that are wrong (ranging from "useless and unclear" to "plain wrong") in your code.
wrt/ the TypeError: as the error message says, your get() function expects two arguments (self and DB) which won't be passed by Flask - and are actually not used in the function anyway. Remove both arguments and you'll get rid of this error - just to find out you now have a NameError on the first line of the get() function (obviously since you didn't import time_statistic nor defined start_date and end_date).

How do you generate the signature for an Azure Blob storage SAS token in Python?

I am trying to build the SAS token required for a blob download URL in Python, following the instructions from MSDN.
My string to sign looks like:
r\n
2016-12-22T14%3A00%3A00Z\n
2016-12-22T15%3A00%3A00Z\n
%2Fblob%2Fmytest%2Fprivatefiles%2F1%2Fqux.txt\n
\n
\n
https\n
2015-12-11\n
\n
\n
\n
\n
_
I've added the newline symbols for clarity and the last line is supposed to be an empty line (with no newline at the end).
The Python method I use for signing the string is:
def sign(self, string):
hashed = hmac.new(base64.b64decode(self.account_key), digestmod=sha256)
hashed.update(string)
base64_str = base64.encodestring(hashed.digest()).strip()
return base64_str
The final URL I build looks like:
https://mytest.blob.core.windows.net/privatefiles/1/qux.txt?sv=2015-12-11&st=2016-12-22T14%3A00%3A00Z&se=2016-12-22T15%3A00%3A00Z&sr=b&sp=r&spr=https&sig=BxkcpoRq3xanEHwU6u5%2FYsULEtOCJebHmupUZaPmBgM%3D
Still, the URL fails with a 403. Any idea on what I am doing wrong?

Update, with the latest storage python library, this is what I used to generate the sas token:
def generate_sas_token(file_name):
sas = generate_blob_sas(account_name=AZURE_ACC_NAME,
account_key=AZURE_PRIMARY_KEY,
container_name=AZURE_CONTAINER,
blob_name=file_name,
permission=BlobSasPermissions(read=True),
expiry=datetime.utcnow() + timedelta(hours=2))
logging.info('https://'+AZURE_ACC_NAME+'.blob.core.windows.net/'+AZURE_CONTAINER+'/'+file_name+'?'+sas)
sas_url ='https://'+AZURE_ACC_NAME+'.blob.core.windows.net/'+AZURE_CONTAINER+'/'+file_name+'?'+sas
return sas_url
Python 3.6 and azure-storage-blob package.

The easiest way to generate SAS token in python is to leverage Azure Storage SDK for Python. Please consider following code snippet:
import time
import uuid
import hmac
import base64
import hashlib
import urllib
from datetime import datetime, timedelta
from azure.storage import (
AccessPolicy,
ResourceTypes,
AccountPermissions,
CloudStorageAccount,
)
from azure.storage.blob import (
BlockBlobService,
ContainerPermissions,
BlobPermissions,
PublicAccess,
)
AZURE_ACC_NAME = '<account_name>'
AZURE_PRIMARY_KEY = '<account_key>'
AZURE_CONTAINER = '<container_name>'
AZURE_BLOB='<blob_name>'
def generate_sas_with_sdk():
block_blob_service = BlockBlobService(account_name=AZURE_ACC_NAME, account_key=AZURE_PRIMARY_KEY)
sas_url = block_blob_service.generate_blob_shared_access_signature(AZURE_CONTAINER,AZURE_BLOB,BlobPermissions.READ,datetime.utcnow() + timedelta(hours=1))
#print sas_url
print 'https://'+AZURE_ACC_NAME+'.blob.core.windows.net/'+AZURE_CONTAINER+'/'+AZURE_BLOB+'?'+sas_url
generate_sas_with_sdk()
Furthermore, to generate SAS token via plain python script, you can refer to the source code at https://github.com/Azure/azure-storage-python/blob/master/azure/storage/sharedaccesssignature.py#L173 for more hints.

Here's an updated code snippet for Python3 and the updated Azure Storage Blob SDK:
from datetime import datetime, timedelta
from azure.storage.blob import (
BlockBlobService,
ContainerPermissions,
BlobPermissions,
PublicAccess,
)
AZURE_ACC_NAME = '<account_name>'
AZURE_PRIMARY_KEY = '<account_key>'
AZURE_CONTAINER = '<container_name>'
AZURE_BLOB='<blob_name>'
block_blob_service = BlockBlobService(account_name=AZURE_ACC_NAME, account_key=AZURE_PRIMARY_KEY)
sas_url = block_blob_service.generate_blob_shared_access_signature(AZURE_CONTAINER,AZURE_BLOB,permission=BlobPermissions.READ,expiry= datetime.utcnow() + timedelta(hours=1))
print('https://'+AZURE_ACC_NAME+'.blob.core.windows.net/'+AZURE_CONTAINER+'/'+AZURE_BLOB+'?'+sas_url)

Based on the documentation (Please see Constructing the Signature String section), the parameters passed to string to sign must be URL decoded. From the link:
To construct the signature string of a shared access signature, first
construct the string-to-sign from the fields comprising the request,
then encode the string as UTF-8 and compute the signature using the
HMAC-SHA256 algorithm. Note that fields included in the string-to-sign
must be URL-decoded.
Please use un-encoded parameter values in your string to sign and that should fix the problem.

Get the Olson TZ name for the local timezone?

How do I get the Olson timezone name (such as Australia/Sydney) corresponding to the value given by C's localtime call?
This is the value overridden via TZ, by symlinking /etc/localtime, or setting a TIMEZONE variable in time-related system configuration files.

I think best bet is to go thru all pytz timezones and check which one matches local timezone, each pytz timezone object contains info about utcoffset and tzname like CDT, EST, same info about local time can be obtained from time.timezone/altzone and time.tzname, and I think that is enough to correctly match local timezone in pytz database e.g.
import time
import pytz
import datetime
local_names = []
if time.daylight:
local_offset = time.altzone
localtz = time.tzname[1]
else:
local_offset = time.timezone
localtz = time.tzname[0]
local_offset = datetime.timedelta(seconds=-local_offset)
for name in pytz.all_timezones:
timezone = pytz.timezone(name)
if not hasattr(timezone, '_tzinfos'):
continue#skip, if some timezone doesn't have info
# go thru tzinfo and see if short name like EDT and offset matches
for (utcoffset, daylight, tzname), _ in timezone._tzinfos.iteritems():
if utcoffset == local_offset and tzname == localtz:
local_names.append(name)
print local_names
output:
['America/Atikokan', 'America/Bahia_Banderas',
'America/Bahia_Banderas', 'America/Belize', 'America/Cambridge_Bay',
'America/Cancun', 'America/Chicago', 'America/Chihuahua',
'America/Coral_Harbour', 'America/Costa_Rica', 'America/El_Salvador',
'America/Fort_Wayne', 'America/Guatemala',
'America/Indiana/Indianapolis', 'America/Indiana/Knox',
'America/Indiana/Marengo', 'America/Indiana/Marengo',
'America/Indiana/Petersburg', 'America/Indiana/Tell_City',
'America/Indiana/Vevay', 'America/Indiana/Vincennes',
'America/Indiana/Winamac', 'America/Indianapolis', 'America/Iqaluit',
'America/Kentucky/Louisville', 'America/Kentucky/Louisville',
'America/Kentucky/Monticello', 'America/Knox_IN',
'America/Louisville', 'America/Louisville', 'America/Managua',
'America/Matamoros', 'America/Menominee', 'America/Merida',
'America/Mexico_City', 'America/Monterrey',
'America/North_Dakota/Beulah', 'America/North_Dakota/Center',
'America/North_Dakota/New_Salem', 'America/Ojinaga',
'America/Pangnirtung', 'America/Rainy_River', 'America/Rankin_Inlet',
'America/Resolute', 'America/Resolute', 'America/Tegucigalpa',
'America/Winnipeg', 'CST6CDT', 'Canada/Central', 'Mexico/General',
'US/Central', 'US/East-Indiana', 'US/Indiana-Starke']
In production you can create such a mapping beforehand and save it instead of iterating always.
Testing script after changing timezone:
$ export TZ='Australia/Sydney'
$ python get_tz_names.py
['Antarctica/Macquarie', 'Australia/ACT', 'Australia/Brisbane',
'Australia/Canberra', 'Australia/Currie', 'Australia/Hobart',
'Australia/Lindeman', 'Australia/Melbourne', 'Australia/NSW',
'Australia/Queensland', 'Australia/Sydney', 'Australia/Tasmania',
'Australia/Victoria']

This is kind of cheating, I know, but getting from '/etc/localtime' doesn't work for you?
Like following:
>>> import os
>>> '/'.join(os.readlink('/etc/localtime').split('/')[-2:])
'Australia/Sydney'
Hope it helps.
Edit: I liked #A.H.'s idea, in case '/etc/localtime' isn't a symlink. Translating that into Python:
#!/usr/bin/env python
from hashlib import sha224
import os
def get_current_olsonname():
tzfile = open('/etc/localtime')
tzfile_digest = sha224(tzfile.read()).hexdigest()
tzfile.close()
for root, dirs, filenames in os.walk("/usr/share/zoneinfo/"):
for filename in filenames:
fullname = os.path.join(root, filename)
f = open(fullname)
digest = sha224(f.read()).hexdigest()
if digest == tzfile_digest:
return '/'.join((fullname.split('/'))[-2:])
f.close()
return None
if __name__ == '__main__':
print get_current_olsonname()

One problem is that there are multiple "pretty names" , like "Australia/Sydney" , which point to the same time zone (e.g. CST).
So you will need to get all the possible names for the local time zone, and then select the name you like.
e.g.: for Australia, there are 5 time zones, but way more time zone identifiers:
"Australia/Lord_Howe", "Australia/Hobart", "Australia/Currie",
"Australia/Melbourne", "Australia/Sydney", "Australia/Broken_Hill",
"Australia/Brisbane", "Australia/Lindeman", "Australia/Adelaide",
"Australia/Darwin", "Australia/Perth", "Australia/Eucla"
you should check if there is a library which wraps TZinfo , to handle the time zone API.
e.g.: for Python, check the pytz library:
http://pytz.sourceforge.net/
and
http://pypi.python.org/pypi/pytz/
in Python you can do:
from pytz import timezone
import pytz
In [56]: pytz.country_timezones('AU')
Out[56]:
[u'Australia/Lord_Howe',
u'Australia/Hobart',
u'Australia/Currie',
u'Australia/Melbourne',
u'Australia/Sydney',
u'Australia/Broken_Hill',
u'Australia/Brisbane',
u'Australia/Lindeman',
u'Australia/Adelaide',
u'Australia/Darwin',
u'Australia/Perth',
u'Australia/Eucla']
but the API for Python seems to be pretty limited, e.g. it doesn't seem to have a call like Ruby's all_linked_zone_names -- which can find all the synonym names for a given time zone.

If evaluating /etc/localtime is OK for you, the following trick might work - after translating it to python:
> md5sum /etc/localtime
abcdefabcdefabcdefabcdefabcdefab /etc/localtime
> find /usr/share/zoneinfo -type f |xargs md5sum | grep abcdefabcdefabcdefabcdefabcdefab
abcdefabcdefabcdefabcdefabcdefab /usr/share/zoneinfo/Europe/London
abcdefabcdefabcdefabcdefabcdefab /usr/share/zoneinfo/posix/Europe/London
...
The duplicates could be filtered using only the official region names "Europe", "America" ... If there are still duplicates, you could take the shortest name :-)

Install pytz
import pytz
import time
#import locale
import urllib2
yourOlsonTZ = None
#yourCountryCode = locale.getdefaultlocale()[0].split('_')[1]
yourCountryCode = urllib2.urlopen('http://api.hostip.info/country.php').read()
for olsonTZ in [pytz.timezone(olsonTZ) for olsonTZ in pytz.all_timezones]:
if (olsonTZ._tzname in time.tzname) and (str(olsonTZ) in pytz.country_timezones[yourCountryCode]):
yourOlsonTZ = olsonTZ
break
print yourOlsonTZ
This code will take a best-guess crack at your Olson Timezone based both on your Timezone Name (as according to Python's time module), and your Country Code (as according to Python's locale module the hostip.info project, which references your IP address and geolocates you accordingly).
For example, simply matching the Timzone Names could result in America/Moncton, America/Montreal, or America/New_York for EST (GMT-5). If your country is the US, however, it will limit the answer to America/New_York.
However, if your country is Canada, the script will simply default to the topmost Canadian result (America/Moncton). If there is a way to further refine this, please feel free to leave suggestions in comments.

The tzlocal module for Python is aimed at exactly this problem. It produces consistent results under both Linux and Windows, properly converting from Windows time zone ids to Olson using the CLDR mappings.

This will get you the time zone name, according to what's in the TZ variable, or localtime file if unset:
#! /usr/bin/env python
import time
time.tzset
print time.tzname

Here's another posibility, using PyICU instead; which is working for my purposes:
>>> from PyICU import ICUtzinfo
>>> from datetime import datetime
>>> datetime(2012, 1, 1, 12, 30, 18).replace(tzinfo=ICUtzinfo.getDefault()).isoformat()
'2012-01-01T12:30:18-05:00'
>>> datetime(2012, 6, 1, 12, 30, 18).replace(tzinfo=ICUtzinfo.getDefault()).isoformat()
'2012-06-01T12:30:18-04:00'
Here it is interpreting niave datetimes (as would be returned by a database query) in the local timezone.

I prefer following a slightly better than poking around _xxx values
import time, pytz, os
cur_name=time.tzname
cur_TZ=os.environ.get("TZ")
def is_current(name):
os.environ["TZ"]=name
time.tzset()
return time.tzname==cur_name
print "Possible choices:", filter(is_current, pytz.all_timezones)
# optional tz restore
if cur_TZ is None: del os.environ["TZ"]
else: os.environ["TZ"]=cur_TZ
time.tzset()

I changed tcurvelo's script to find the right form of time zone (Continent/..../City), in most of cases, but return all of them if fails
#!/usr/bin/env python
from hashlib import sha224
import os
from os import listdir
from os.path import join, isfile, isdir
infoDir = '/usr/share/zoneinfo/'
def get_current_olsonname():
result = []
tzfile_digest = sha224(open('/etc/localtime').read()).hexdigest()
test_match = lambda filepath: sha224(open(filepath).read()).hexdigest() == tzfile_digest
def walk_over(dirpath):
for root, dirs, filenames in os.walk(dirpath):
for fname in filenames:
fpath = join(root, fname)
if test_match(fpath):
result.append(tuple(root.split('/')[4:]+[fname]))
for dname in listdir(infoDir):
if dname in ('posix', 'right', 'SystemV', 'Etc'):
continue
dpath = join(infoDir, dname)
if not isdir(dpath):
continue
walk_over(dpath)
if not result:
walk_over(join(infoDir))
return result
if __name__ == '__main__':
print get_current_olsonname()

This JavaScript project attempts to solve the same issue in the browser client-side. It works by playing "twenty questions" with the locale, asking for the UTC offset of certain past times (to test for summer time boundaries, etc.) and using those results to deduce what the local time zone must be. I am not aware of any equivalent Python package unfortunately, so if someone wanted to use this solution it would have to be ported to Python.
While this formula will require updating every time (at worst) the TZ database is updated, a combination of this algorithm and the solution proposed by Anurag Uniyal (keeping only possibilities returned by both methods) sounds to me like the surest way to compute the effective local timezone. As long as there is some difference between the UTC offset of at least one local time in any two time zones, such a system can correctly choose between them.