I am trying to extract the specific details like authored_date from the attribute which I am getting with help of python.
What is my end goal:
I want to extract the specific branches which are named as tobedeleted_branch1, tobedeleted_branch2 and delete them with help of my script if the authored_date is more than 7 days.
I am beginner in this and learning this currently.
So, what I want to do is,
Extract the authored date from the output and check if it is older than 7 days.
If it is older than 7 days I will go ahead and perform whatever I want to perform in if condition.
import gitlab, os
#from gitlab.v4.objects import *
# authenticate
TOKEN = "MYTOKEN"
GITLAB_HOST = 'MYINSTANCE' # or your instance
gl = gitlab.Gitlab(GITLAB_HOST, private_token=TOKEN)
# set gitlab group id
group_id = 6
group = gl.groups.get(group_id, lazy=True)
#get all projects
projects = group.projects.list(include_subgroups=True, all=True)
#get all project ids
project_ids = []
for project in projects:
# project_ids.append((project.path_with_namespace, project.id, project.name ))
project_ids.append((project.id))
print(project_ids)
for project in project_ids:
project = gl.projects.get(project)
branches = project.branches.list()
for branch in branches:
if "tobedeleted" in branch.attributes['name']:
print(branch.attributes['name'])
#print(branch)
print(branch.attributes['commit'])
#branch.delete()
The output which I am getting from printing the print(branch.attributes['commit']) is like :
{'id': '1245829930', 'short_id': '124582', 'created_at': '2021-11-15T09:10:26.000+00:00', 'parent_ids': None, 'title': 'branch name commit' into \'master\'"', 'message': 'branch name commit', 'author_name': 'Administrator', 'author_email': 'someemail#gmail.com', 'authored_date': '2021-11-15T09:10:26.000+00:00', 'committer_name': 'Administrator', 'committer_email': 'someemail#gmail.com', 'committed_date': '2021-11-15T09:10:26.000+00:00', 'trailers': None, 'web_url': 'someweburl'}
From the above output, I want to extract the 'authored_date' and check if it is greated than 7 days I will go ahead and delete the merged branch.
Any help regarding this is highly appreciated.
from datetime import datetime
def get_day_diff(old_date):
old_datetime = datetime.fromisoformat(old_date)
# Check timezone of date
tz_info = old_datetime.tzinfo
current_datetime = datetime.now(tz_info)
datetime_delta = current_datetime - old_datetime
return datetime_delta.days
# test with authored_date
authored_date = '2021-11-15T09:10:26.000+00:00'
if get_day_diff(authored_date) > 7:
# then delete branch
branch.delete()
import datetime
created_at = '2021-11-15T09:10:26.000+00:00'
t = datetime.datetime.fromisoformat(created_at)
n = datetime.datetime.now(tz=t.tzinfo)
if n - t <= datetime.timedelta(days=7):
... # do someting
using your branch.attributes about your commit inside your loop, you can do the following (make sure you import datetime). You'll want to append the branch name to be deleted to a list that you'll iterate over after, as you do not want to modify any object that you are currently iterating over (IE deleting items from the branches object that you are still iterating over).
from datetime import datetime
...
branches_to_delete = []
for project in project_ids:
project = gl.projects.get(project)
branches = project.branches.list()
for branch in branches:
if "tobedeleted" in branch.attributes['name']:
print(branch.attributes['name'])
#print(branch)
print(branch.attributes['commit'])
branch_date_object = datetime.strptime((branch.attributes['commit']['authored_date'].split('.')[0]), "%Y-%m-%dT%H:%M:%S")
days_diff = datetime.now() - branch_date_object
if days_diff.days > 7:
branches_to_delete.append(branch.attributes['name'])
for branch in branches_to_delete:
# perform your delete functionality
Related
I'm trying to write a python script using boto3 in order to get hourly prices of an instance, given the instance ID. I should remark that I'm not speaking about costs that you can get from cost explorer, I'm speaking about nominal hourly price, for example for an 'ec2' instance.
I've already found some examples using "boto3.client('pricing',...)" and a bunch of parameters and filters as in:
https://www.saisci.com/aws/how-to-get-the-on-demand-price-of-ec2-instances-using-boto3-and-python/
which also requires region code to region name conversion.
I would like not to have to specify every instance detail and parameter for that query.
Can anybody help me to find a way to get that info just having the ec2 instance ID?
Thanks in advance.
You have to pass all that information. If you want to write a script that takes an instance ID and returns the hourly price, then you would first need to use the instance ID to lookup the instance details, and then pass those details to the pricing query.
You have to specify most of the information but not all of it.
For example, region_name is optional if you:
Have configured AWS CLI on the machine on which your Python script is running (ie. ~/.aws/config is present and the region is configured).
OR
You are running the Python script on an AWS resource that has a role attached to it with a policy that allows you to retrieve the spot pricing information.
For example, I am able to run this script that retrieves my current spot instances and gets their current hourly cost, and calculates a bid price for me based on the spot price history for that particular instance type without specifying the region anywhere:
#!/usr/bin/env python3
import boto3
import json
from datetime import datetime
from datetime import timedelta
from collections import namedtuple
def get_current_pricing():
pricing = []
ec2_client = boto3.client('ec2')
ec2_resource = boto3.resource('ec2')
response = ec2_client.describe_spot_instance_requests()
spot_instance_requests = response['SpotInstanceRequests']
for instance_request in spot_instance_requests:
if instance_request['State'] == 'active':
instance = ec2_resource.Instance(instance_request['InstanceId'])
for tag in instance.tags:
if tag['Key'] == 'Name':
application = tag['Value']
break
price = {
'application': application,
'instance_type': instance_request['LaunchSpecification']['InstanceType'],
'current_price': float(instance_request['SpotPrice']),
'bid_price': get_bid_price(instance_request['LaunchSpecification']['InstanceType'])
}
pricing.append(price)
return pricing
def get_bid_price(instancetype):
instance_types = [instancetype]
start = datetime.now() - timedelta(days=1)
ec2 = boto3.client('ec2')
price_dict = ec2.describe_spot_price_history(
StartTime=start,
InstanceTypes=instance_types,
ProductDescriptions=['Linux/UNIX']
)
if len(price_dict.get('SpotPriceHistory')) > 0:
PriceHistory = namedtuple('PriceHistory', 'price timestamp')
for item in price_dict.get('SpotPriceHistory'):
price_list = [PriceHistory(round(float(item.get('SpotPrice')), 5), item.get('Timestamp'))]
price_list.sort(key=lambda tup: tup.timestamp, reverse=True)
bid_price = round(float(price_list[0][0]), 5)
leeway = round(float(bid_price / 100 * 10), 5)
bid_price = round(float(bid_price + leeway), 5)
return bid_price
else:
raise ValueError(f'Invalid instance type: {instancetype} provided. '
'Please provide correct instance type.')
if __name__ == '__main__':
current_pricing = get_current_pricing()
print(json.dumps(current_pricing, indent=4, default=str))
i am developing a 3rd party tool which is fetching report data from apis, i am indexing this in elastic search using
django_elasticsearch_dsl
But every time i index the fetched data it overrides the previous one.
one approach was mentioned in the documentation help
i.e We can store index fer day with date as suffix in index name.
For example:
index_name = report-yyyymmdd-hhmmss
How to do this in python using django_elasticsearch_dsl?
What is the best practice to do this?
Here is my code:
class ReportDocument(Document):
def prepare_labels(self, instance):
if instance.labels == '--':
return ''
return list(map(str.strip, eval(instance.labels)))
class Index:
name = "report1"
settings = {
'number_of_shards': 1,
'number_of_replicas': 0
}
class Django:
model = Report
fields = [
field.name for field in Report._meta.get_fields()]
auto_refresh = False
I'm trying to retrieve data from mongodb via mongoengine within a specified time span. Below is the db model used.
class DeviationReport(db.Document):
meta = {'collection': 'DeviationReport'}
created_at = db.DateTimeField()
date = db.DateTimeField()
author = db.StringField()
read_by = db.ListField(default=[])
prod_line = db.ReferenceField(ProductionLine)
product = db.ReferenceField(Product)
description = db.StringField()
What I've tried is the code below. It does however not return any results. I've used a similar approach when I've needed to build dynamic queries depending on user input.
kwargs = {}
start = datetime.datetime(2018, 12, 11)
end = datetime.datetime(2019, 03, 13)
kwargs['created_at'] = { '$lt': end, '$gt': start }
DeviationReport.objects(**kwargs)
I've obviously made sure that there are objects within the date range, and I've read other similar posts where the query below has been successfully used. How do I get my query to return everything between 'start' and 'end', or how do I rewrite it to do as I wish?
Thanks you.
I worked around/solved the problem by first getting my results sans date filtering using **kwargs and then filtered that using Q. It may not be optimal, but it works for what I need it to do.
reports = DeviationReport.objects(**kwargs)
reports = reports.filter((Q(date__gte=start) & Q(date__lte=end)))
There's a number of ways to achieve your query, adjust the collection and params accordingly using the example below:
date_to = datetime.datetime.utcnow() # The end date
date_from = date_to - datetime.timedelta(days=120) # The start date
query_a = Application.objects(category="rest_api").filter(
date_created__gte=date_from,
date_created__lte=date_to
)
query_b = Application.objects(
date_created__gte=date_from,
date_created__lte=date_to
).filter(category="rest_api")
query = {"category": "rest_api"}
query_c = Application.objects(
date_created__gte=date_from,
date_created__lte=date_to,
**query
)
Querying with Q as suggested above did not work for me, but a raw query did:
raw_query = {'date': {'$gte': start, '$lt': end}}
reports = DeviationReport.objects(__raw__=raw_query)
Given some code like this:
# coding: utf-8
import datetime
from django.db import models
from django.contrib.auth.models import User
from django.contrib.sites.models import Site
class Premium(models.Model):
"""Access to Premium Features™®."""
end = models.DateField()
user = models.ForeignKey(User)
site = models.ForeignKey(Site)
def get_ending_premiums():
"""Get a queryset of all Premiums for which a user has none following."""
tomorrow = datetime.date.today() + datetime.timedelta(days=1)
future_premiums = Premium.objects.filter(end__gt=tomorrow).values('user', 'site')
return Premium.objects.filter(end=tomorrow).exclude(
# Would love if something like this actually worked...
user_and_site__in=future_premiums,
)
How can I complete get_ending_premiums()? One of the key things is I want Premiums only when there isn't another one that ends later, but on a per-site basis. So if a user has another Premium on groceries.com, the one about to end tomorrow doesn't get returned, but if they don't have another Premium on officesupplies.com, that one does get returned.
(Note the line with with the comments before it doesn’t actually work... that’s the part I need to complete.)
I can work out how to do this outside the ORM but I’d really prefer an ORM solution, as we’re planning on switching database vendors in a few months, so I’m trying to avoid raw SQL as much as possible.
Here’s a test for the behavior I’d like to get:
class PremiumTest(TestCase):
def test_gets_ending_premiums(self):
today = date(2020, 6, 5)
tomorrow = today + timedelta(days=1)
next_year = today + timedelta(days=366)
groceries = Site.objects.create(domain='groceries.com')
catvids = Site.objects.create(domain='catvids.com')
dave = User.objects.create_user('dave')
sally = User.objects.create_user('sally')
Premium.objects.create(user=dave, site=groceries, end=tomorrow)
Premium.objects.create(user=dave, site=groceries, end=next_year)
Premium.objects.create(user=dave, site=catvids, end=tomorrow)
Premium.objects.create(user=sally, site=groceries, end=tomorrow)
Premium.objects.create(user=sally, site=catvids, end=tomorrow)
Premium.objects.create(user=sally, site=catvids, end=next_year)
ending_premiums = get_ending_premiums(today)
ending = set((p.user, p.site) for p in ending_premiums)
self.assertNotIn((dave, groceries), ending)
self.assertIn((dave, catvids), ending)
self.assertIn((sally, groceries), ending)
self.assertNotIn((sally, catvids), ending)
self.assertEqual(2, len(ending_premiums))
I've come up with this... It's got some raw SQL but it still returns a QuerySet with normal QuerySet methods (although it uses the apparently deprecated QuerySet.extra() method)
def get_ending_premiums(day=None):
"""Get a queryset of Premiums for which a user has none following."""
if day is None:
day = date.today()
tomorrow = day + timedelta(days=1)
ending_premiums = Premium.objects.filter(
end=tomorrow,
).extra(
where=['NOT EXISTS (SELECT NULL FROM premium_premium child where premium_premium.site_id = site_id AND premium_premium.user_id = user_id AND end > %s )'],
params=[tomorrow],
)
return ending_premiums
Still wondering if there isn’t a better way...
Currently working on a project with TurboGears2 and ToscaWidgets2. I have a form setup with a few static fields, name, date, and point of contact information. Inside this form I have added a sub form where the user can dynamically add numerous entries in a GrowingGridLayout. The form, its layout, and submitting information is all well and good but I'm having a hard time figuring out how to capture the information from the GrowingGridLayout once it's passed on for saving. Guess the main points are, how do I know how many entries were included in the form?
Included the code for the form:
class OnrampForm(twf.Form):
title = "Onramp Submission Form"
class child(twd.CustomisedTableForm):
onramp_name = twf.TextField(validator=twc.Required)
class Destinations (twd.GrowingGridLayout):
environment = twf.SingleSelectField(label='Environment', validator=twc.Validator(required=True), options=[<OPTIONS>])
location = twf.SingleSelectField(validator=twc.Required, label='Location', options=[<OPTIONS>])
jms_type = twf.SingleSelectField(label='JMS Type', validator=twc.Validator(required=True), options=[<OPTIONS>])
subscription_type = twf.SingleSelectField(label='Subscription Type', validator=twc.Validator(required=True), options=[<OPTIONS>])
onramp_status = twf.SingleSelectField(prompt_text='Status', options=['Initial Release', 'Update'], validator=twc.Required)
current_date = datetime.date.today()
need_by_date = twd.CalendarDatePicker(validators=[twc.Required, twc.DateTimeValidator])
need_by_date.default = current_date + datetime.timedelta(days=30)
organization = twf.TextField(validator=twc.Required)
poc_name = twf.TextField(validator=twc.Required)
poc_email = twf.EmailField(validator=twc.EmailValidator)
poc_phone = twf.TextField(validator=twc.Required)
poc_address = twf.TextField()
poc_city = twf.TextField()
poc_state = twf.TextField()
onramp_form = twf.FileField()
submit = twf.SubmitButton(value="Submit")
action = "/print_args"
submit = ""
If you controller #validates against the form you should get the data into the Destination parameter which should be a list of dictionaries.
Also I just noticed you have two nested forms, that's something that might confuse TW2 pretty much. What you wanted to do is probably have OnrampForm inherit CustomisedForm and then have child inherit TableLayout. See http://turbogears.readthedocs.org/en/latest/cookbook/TwForms.html#displaying-forms
PS: note that need_by_date.default = current_date + datetime.timedelta(days=30) will always return 30 days from when the server started as you are actually storing a current_date = datetime.date.today() class variable that gets computed when the module is imported and no more.
You should use default = Deferred(lambda: datetime.date.today() + datetime.timedelta(days=30)) to achieve that