I have an existing Devops pipeline that trains ML models. To guaranty the robustness for the models, it will be necessary to retrain it periodically. For this I decided to create an Azure function that will be executed each month. will It collect the new data, apply the data pre-processing and finally trigger the Azure Devops training pipeline. All of this must be done with python.
By doing a research, I understood that this can be done using REST Azure Devops API request.
I founded this python git repo https://github.com/microsoft/azure-devops-python-api which provides an API to communicate with Azure DevOps. I executed the code provided by this package which displays the list of my devops projects. But I can't found how trigger the pipeline.
Assuming that my organisation named ORGA1, the project named PROJ1 and the pipeline that I want ti execute named PIPELINE1, How can I launch it using an Azure function or even a simple python script ?
PS: I am using a python 3.9 Timer Trigger Azure function.
Thank you in advance for your help.
EDIT
I tired to use LOGIC APP to do this like #mohammed described in the comment and I think that this is a good solution. Above the workflow that I created:
So I launch the logic app each X hours, this will trigger the azure Devops, and ASA it end training with Success it will send me an email.
I have one error here, is that I am creating a new release and not triggering a specific pipeline each time. But navigating in the different actions under the devops service, I cannot found any thing related to launching a devops pipeline. Can anyone have an idea how to do it.
You can use a Logic App with a timer to trigger your DevOps pipeline instead of an Azure function, as it has all the built-in connectors required to interface with your DevOps. See : https://www.serverlessnotes.com/docs/keep-your-team-connected-using-azure-devops-and-azure-logic-apps
Thanks to the tips provided by #Mohammed I found a solution. Logic App provides what I am looking for. Under the list of Devops connectors provided the by the Logic App, there is a connector named Queue a new build and this is exactly what I am looking for. This is my first experiment architecture and I will update it later by adding Azure Function service before calling the Devops Pipeline.
You may try using Azure Durable Functions, you can kind of replicate what a Logic App does while still using Azure Functions. See documentation here 1
Related
Is it possible to call ADF pipelines from external schedulers? We have an enterprise scheduler, and want to integrate ADF scheduler with the same. Is it possible?
Is it good design to call a Python/PowerShell script from the enterprise scheduler, which will trigger the adf pipeline.
Sure they can! In fact, the docs explicitly tell how:
You can manually run your pipeline by using one of the following methods:
.NET SDK
Azure PowerShell module
REST API
Python SDK
I am trying to create an azure ci/cd pipeline for my python application. I have tried many ways but not get success. I can create CI successfully even in some cases cd also but not able to see the output on the azure app service.
I use Linux app services that use the Python 3.7 version.
I can create ci-cd successfully using the YAML file but I want to create using the classic editor without YAML, as I have some restrictions using yaml.
I will post the steps I deploy a simple hello world project with DevOps CI/CD pipeline.
1. Create pipeline:
2. Create Release pipeline:
3. Save and queue your pipeline, the release pipeline would be triggered. Here is the file structure on Azure KUDU:
I'm looking to create a publisher that streams and sends tweets containing a certain hashtag to a pub/sub topic.
The tweets will then be ingested with cloud dataflow and then loaded into a Big Query database.
In the following article they do something similar where the publisher is hosted on a docker image on a Google Compute Engine instance.
Can anyone recommend alternative Google Cloud resources that could host the publisher code more simply, that avoids the need to create a docker file etc?
The publisher would need to run constantly. Would cloud run for e.g. be a suitable alternative?
There are some workarounds I can think of:
A quick way to avoid containers architecture is having the on_data method inside a loop, for example, by using something like while(true) or start a Stream like explained in Create your Python script and run the code in a Compute Engine in the background with nohup python -u myscript.py. Or follow the steps described in Script on GCE to capture tweets that uses tweepy.Stream to start the streaming.
You might want to reconsider the Dockerfile option since its configuration could be not so difficult, see Tweets & pipelines where there is a script that read the data and publish to PubSub, you will see that 9 lines are used for the Docker file and it is deployed in App Engine using Cloud Build. Another implementation with a Docker file that requires more steps is twitter-for-bigquery, in case it helps, you will see that there are more specific steps and more configurations.
Cloud Functions is also another option, in this guide Serverless Twitter with Google Cloud you can check the Design section to know if it fits your use case.
Airflow with Twitter Scraper could work for you since Cloud Composer is a managed service for Airflow and you can create an Airflow environment quickly. It uses the Twint library, check the Technical section in the link for more details.
Stream Twitter Data into BigQuery with Cloud Dataprep is a workaround that put aside complex configurations. In this case the job won't run constantly but can be scheduled to run within minutes.
I am a newbie who wants to deploy his flask app using google cloud functions. When I am searching it online, people are telling me to deploy it as a Flask app. I want to ask if there is any difference between those two.
A cloud instance or deploying flask app on google cloud VS cloud serverless function
As described by John and Kolban, Cloud Functions is a single purpose endpoint. You want to perform 1 thing, deploy 1 function.
However, if you want to have a many consistent things, like a microservice, you will have to deploy several endpoints that allow you to perform a CRUD on the same data object. You should prefer to deploy several endpoints (CRUD) and to have the capability to easily reuse class and object definitions and business logic. For this, a Flask webserver is that I recommend (and I prefer, I wrote an article on this).
A packaging in Cloud Run is the best for having a serverless platform and pay-per-use pricing model (and automatic scaling and...).
There is an additional great thing: Cloud Functions request object is based on Flask request object. By the way, and it's that I also present in my article, it's easy to switch from one platform to another one. You only have to choose according with your requirements, your skills,... I also wrote another article on this
If you deploy your Flask app as an application in a Compute Engine VM instance, you are basically configuring a computer and application to run your code. The notion of Cloud Functions relieves you from the chore and toil of having to create and manage the environment in which your program runs. A marketing mantra is "You bring the code, we bring the environment". When using Cloud Functions all you need do is code your application logic. The maintenance of the server, scaling up as load increases, making sure the server is available and much more is taken care of for you. When you run your code in your own VM instance, it is your responsibility to manage the whole environment.
References:
HTTP Functions
Deploying a Python serverless function in minutes with GCP
I am working on extracting Features from csv files and I use Python to perform the task. I am inside Azure and created a Python Application using Visual studio 2017. It works perfectly fine and i am looking for ways to automate the process so that it runs in batches per schedule.
I dont want to post it as a web job because the script has some references to the file in local disk of my VM. Could some one tell me the options available to run this solution in batch?
According to your description , I provide you with several ways as below to run your solution in batch.
1.Web Job
Actually , you can package Python script-dependent modules or references together and send them to webjob. Then you can find their absolute path on the KUDU and refer to them in your script, so this does not affect your use of webjob. For this process you can refer to the case I used to answer :Python libraries on Web Job.
Please note that Web Job can be executed per second at least.
2.Azure Scheduler
Azure Scheduler allows you to declaratively describe actions to run in the cloud. It then schedules and runs those actions automatically. You can periodically call your app script url. More details , please refer to the official tutorial.
Please note that Azure Scheduler can be executed per minute at least.
3.Azure Functions
Like the previous method , you can use Azure functions timer trigger to periodically call your app script url. More details , please refer to the official tutorial.
4.Azure Batch
Azure Batch schedules compute-intensive work to run on a managed collection of virtual machines, and can automatically scale compute resources to meet the needs of your jobs.Considering that Azure Batch is used in large data operations, the cost of combining your situation is relatively high and I don't suggesting you use.More details , please refer to the official tutorial.
Hope it helps you.