I need help with celery and python. I was able to configure celery after a lot of trouble. I got help from this link https://github.com/larsbutler/celery-examples I tried the example program and it ran fine as long as all the tasks.py, demo.py and celeryconfig.py were on the same folder. I transferred demo.py to some other folder and tried running it. It threw ImportError: Unable to import tasks module. I am using rabbitmq. I am not able to find more resources for python using celery.
I have two systems - Alice and Bravo. I want to access the function add residing in Alice from Bravo. I need help with the configuration of celery for this purpose. It is like client and server architecture.
FYI, the OP contacted me directly last Friday (duplicate post? :P) and I was able to help him resolve the issue. I think we can consider this question resolved.
As S. Lott mentioned, the proper solution was to add the other folder (containing demo.py) to PYTHONPATH.
Related
So I'm fairly new to airflow and have only really been using github as a fairly basic push/ pull tool rather than getting under the hood and using it for anything more complex.
That being said, now is the time I wish to do something more complex with airflow/ github.
My organisation uses Google cloud for pretty much everything and I currently use magnus to trigger my scheduled queries.
For many reasons, I'm aiming to move over to airflow to perform these tasks however what I'm actually trying to do is host my source code in github and use gitpython to find the .sql files for airflow to then trigger my refresh.
I'm seemingly having trouble understanding how I can possibly 'host' my github repo in an airflow instance and then isolate a file to push to a dag task.
So, problem 1 - each time I try and connect to my remote repo, I receive a windows error
Cmd('git') not found due to: FileNotFoundError('[WinError 2] The system cannot find the file specified')
cmdline: git pull Remote_server_Address.git
I'm trying various commands but not really finding the documentation useful.
As I'm aiming to host the repo in airflow (preferably within just a python instance) I'm hoping I don't need to provide a local path - but even when I try to do so, I still get the same error.
All help appreciated and apologies if it's vague.
Any other integration suggestions would also be recommended.
Thanks
It is a little hard to understand the setup you describe.
For example
isolate a file to push to a dag task
Does this mean you want a task read a specific file when you run an instance of it?
If that is the case you probably want to pass the file location (likely hosted in GCS) to the dag. This explains how.
However, a more common pattern is for something like a daily job to automatically select the file or run a query based on the date.
You could also setup a sensor that will trigger a dag when a file is added to a specific GCS folder using the gcs sensor.
It seems that flask-crontab cannot run on my machine because there is no such thing as a windows cron.
This is the error I get: ModuleNotFoundError: No module named 'fcntl'
If this is the case, how can I write a scheduled job in a Flask application?
Specifically, I need to be able to deploy the application--so it won't be running on windows once it goes into production, but in the meantime, I can't debug if I can't test the cronjob on my own computer.
Any help is appreciated, e.g. pointing me to documentation that is useful, suggesting other extensions, etc. Thanks!
Pete
I would recommend you start using Celery,
Celery is a great library to use for job scheduling, whether you want this job to be periodic or upon request tasks.
Follow this guide
You can use Apache Airflow. It is written in Python. Its documentation is here.
After several weeks looking for some information here and google, I've decided to post it here to see if anyone with the same problem can raise me a hand.
I have a java application developed in Eclipse Ganymede using tomcat to connect with my local database. The problem is that I want to send a simple message ("Hello World") to a Kafka Topic published on a public server. I've imported the libraries and developed the Kafka function but something happens when I run in debug mode. I have no issues or visible errors when compiling, but when I run the application and push the button to raise this function it stops in KafkaProducer function because there is NoClassDefFoundError kafka.producer..... It seems like it is not finding the library properly, but I have seen that it is in the build path properly imported.
I am not sure if the problem is with Kafka and the compatibility with Eclipse or Java SDK (3.6), it could be?. Anyone knows the minimum required version of Java for Kafka?
Also, I have found that with Kafka is really used Scala but I want to know if I can use this Eclipse IDE version for not change this.
Another solution that I found is to use a Python script called from the Java application, but I have no way to call it from there since I follow several tutorials but then nothing works, but I have to continue on this because it seems an easier option. I have developed the .py script and works with the Kafka server, now I have to found the solution to exchange variables from Java and Python. If anyone knows any good tutorial for this, please, let me know.
After this resume of my days and after hitting my head with the walls, maybe someone has found this error previously and can help me to find the solution, I really appreciate it and sorry for the long history.
Please include the Kafka client library within the WAR file of the Java application which you are deploying to Tomcat
Please use org.apache.kafka.clients.producer.KafkaProducer rather than kafka.producer.Producer (which is the old client API) and make sure you have the Kafka client library on the classpath. The client library is entirely in Java. It's the old API that's written in scala, as is the server-side code. You don't need to import the server library in your code or add it to the classpath if you use the new client API.
At the end the problem was related with the library that was not well added. I had to add it in the build.xml file, importing here the library. Maybe this is useful for the people who use an old Eclipse version.
So now it finds the library but I have to update Java version, other matter. So it is solved
I've never worked with Django before so forgive me if a question sounds stupid.
I need to develop a web application, but I do not want to deploy it on a server. I need to package it, so that others would "install" it on their machine and run it. Why I want to do it this way? There are many reasons, which I don't want to go into right now. My question is: can I do it? If yes, then how?
This is possible. However, the client machine would need to be equipped with the correct technologies for this to work.
When you launch a web app on a server (live), the server is required to have certain settings and installs. For example, a Django web app: the server must have a version of Django installed.
Hence, whichever machine is running your web app, must have Django installed. It presumably also needs to have the database too. It might be quite a hassling process but it's possible.
Just like as a developer, you may have multiple users working on 1 project. So, they all need to have that project 'installed' on their devices so they can run it locally.
You need to either use a python to executable program, with Django already in it. The website files you can place into the dist folder or whatever folder has the executable in it. Then you can compress it and share it with others (who have the same OS as you).
For an example:
You have this script in Django (I'm too lazy to actually write one), and you want to share it with someone who doesn't have Python and Django on his/her computer.
I am a python programmer, and server administration was always a bit hard for me to immerse to. I always read tutorials and in practice just repeated the steps each time I set up a new project. I always used uwsgi, but realized that gunicorn is easier for simple projects like mine.
Recently, I successfully set up my first single gunicorn application with the help of this article: https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-16-04
But what should I do if I want to launch another app with gunicorn? Should I just make another systemd service file, like myproject.service? I'm looking for convenient 'one click' setup, so I can easily transfer my project to another machine, or add more gunicorn applications without much configuration. Or maybe, I should use another process manager like supervisor? What is the best solution for a newbie like me?
Sorry if my question's too dumb, but I'm really trying.
Thank you!