I have python code that depends on specific libraries like selenium and interaction with google chrome to extract data from the web.
my code works fine but i need a lot of records to do analysis, so i can't leave my computer on, to run the script for a month.
That's why I thought of running the script in a cloud service like aws but I don't have a clear idea of how to do it, because I need the script to not stop
and I would rather not have to pay for it (or at least not that much money)
That said, my code opens a website, looks for a specific text data and saves it in a csv document.
I thank you in advance for the help
You will have to check the terms of each cloud service as many do have downtime/restarts on their free tiers.
The kind of task you're describing shouldn't be very resource hungry, so you may be better off setting up your own server using a Raspberry Pi or similar.
Related
First I'd like to say that I know similar questions about calling Python code in Flutter have been asked before, but I think this particular case has some challenges.
Some notes about the app I'm aiming for:
Basically a note taking app, records a lecture or meeting or whatever and transcribes the text for you, with a few extra features thrown in. I'd like to have all speech being processed locally both to ensure it works offline and reduce the app's dependence on cloud services.
I'm trying to use Whisper, a new speech to text software that processes everything locally, which is a necessity for my app. I know I could make a Flutter plugin but I'm not sure if that's the best route to go about this for a few reasons:
I haven't done it before, so it would be quite a time investment to do this and just hope it works out.
One of the ways I've seen of doing this involves sending data over http between Python and Flutter, but Whisper would need a continuous stream of audio to work properly which I'm not sure this approach is suited for.
I'd really like to have 1 codebase that runs on any device.
I'd be fine with the app only working on pc for now, but I'd like to also have it working on Android and maybe IOS if reasonably possible. Any other routes I can take towards development are great too but I'd really like to stick with Flutter for this app if I can.
Just found that one: https://github.com/azkadev/whisper_dart
Did not tried it until now but seems to be worth the try.
I work as a DS in a ver small company, so all of the DS team is very "young" in this field.
We are currently experiencies issues with cooperation, especific at the writing code moment.
We've tried with VScode live share which is a great extension but, due to our pc's limitations goes hard to work when we are working with big df.
I was looking over deepnote, which sounds really great, but is has no support with MSsql server.
so, any alternative? Also we're thinking in cloud migration, like azure or AWS, but I was unable to find a proper way to do it or if there we can work in real time co-editting
so, any help or advice?
Ok, so I'm a total noob with aspirations of learning to code. I've read about a guy who, for example, wrote a script which, if he was at work past a certain time, would automatically send a text to his wife stating he would be late. I want to do something sorta similar.
What I want in essence is a script that will log in to a website at a certain time of day, check if a box/text is green/yes or red/no, and send a text or notification to my phone informing me of the result each day.
The progress I've made so far is installing Python, installing PyCharm and done some research about tools I could use toward achieving my goal. Selenium seems like it would be capable of logging into the website, but I've no idea how to go about setting up a conditional statement to check the result, nor how I could set it up to send a text/notification to my phone. Also, if there is a more appropriate tool I should look into rather than Selenium and Python, I'm not attached to the idea of using these specific tools.
Finally, I realize that this may end up being too complicated for a first project, so I'd be up for hiring a freelancer to set this up. Equally, if this is something that could feasibly be written by someone with very little knowledge of coding such as myself, I'd really appreciate some direction from an expert!
Thanks for any input!
You are on the right track with selenium for web form automation. Sending notification however would require something else as was pointed out, and if you're on windows you can use windows task scheduler to automate, to performed only on certain time of day etc.
To make things more simplified, you can also look up general purpose automation programs that might support all these features together. For example, JRVSInputs uses selenium for web auto-fills https://jrvs.in/forums/viewtopic.php?t=182 and have features to send email or windows notifications. It can convert all its scripts into a neat batch file, you can then automate this batch file in the task scheduler.
Okay, so basically I am creating a website. The data I need to display on this website is delivered twice daily, where I need to read the delivered data from a file and store this new data in the database (instead of the old data).
I have created the python functions to do this. However, I would like to know, what would be the best way to run this script, while my flask application is running? This may be a very simple answer, but I have seen some answers saying to incorporate the script into the website design (however these answers didn't explain how), and others saying to run it separately. The script needs to run automatically throughout the day with no monitoring or input from me.
TIA
Generally it's a really bad idea to put a webserver to handle such tasks, that is the flask application in your case. There are many reasons for it so just to name a few:
Python's Achilles heel - GIL.
Sharing system resources of the application between users and other operations.
Crashes - it happens, it could be unlikely but it does. And if you are not careful, the web application goes down along with it.
So with that in mind I'd advise you to ditch this idea and use crontabs. Basically write a script that does whatever transformations or operations it needs to do and create a cron job at a desired time.
I have just coded a trading algorithm and some analytics software for the stock market which in itself works fine.
Since my computer is not always running or internetconnection is not running perfectly I would like to source the script out and put it on a Webserver for example, where it would run all day and night.
Do you guys now I could do that?
I would also like to build a user interface using django to monitor live performance.
Does anybody know what would be necessary to implement these steps?
Thanks in advance and kind regards
Marcel Kresse
This is very general question and the answer is close to "sky is the limit". As mentioned above, any cloud service provider will do.
Most (if not all) clouds have dedicated images for web servers and Django deployments. Have fun.