I work as a DS in a ver small company, so all of the DS team is very "young" in this field.
We are currently experiencies issues with cooperation, especific at the writing code moment.
We've tried with VScode live share which is a great extension but, due to our pc's limitations goes hard to work when we are working with big df.
I was looking over deepnote, which sounds really great, but is has no support with MSsql server.
so, any alternative? Also we're thinking in cloud migration, like azure or AWS, but I was unable to find a proper way to do it or if there we can work in real time co-editting
so, any help or advice?
Related
First I'd like to say that I know similar questions about calling Python code in Flutter have been asked before, but I think this particular case has some challenges.
Some notes about the app I'm aiming for:
Basically a note taking app, records a lecture or meeting or whatever and transcribes the text for you, with a few extra features thrown in. I'd like to have all speech being processed locally both to ensure it works offline and reduce the app's dependence on cloud services.
I'm trying to use Whisper, a new speech to text software that processes everything locally, which is a necessity for my app. I know I could make a Flutter plugin but I'm not sure if that's the best route to go about this for a few reasons:
I haven't done it before, so it would be quite a time investment to do this and just hope it works out.
One of the ways I've seen of doing this involves sending data over http between Python and Flutter, but Whisper would need a continuous stream of audio to work properly which I'm not sure this approach is suited for.
I'd really like to have 1 codebase that runs on any device.
I'd be fine with the app only working on pc for now, but I'd like to also have it working on Android and maybe IOS if reasonably possible. Any other routes I can take towards development are great too but I'd really like to stick with Flutter for this app if I can.
Just found that one: https://github.com/azkadev/whisper_dart
Did not tried it until now but seems to be worth the try.
I have python code that depends on specific libraries like selenium and interaction with google chrome to extract data from the web.
my code works fine but i need a lot of records to do analysis, so i can't leave my computer on, to run the script for a month.
That's why I thought of running the script in a cloud service like aws but I don't have a clear idea of how to do it, because I need the script to not stop
and I would rather not have to pay for it (or at least not that much money)
That said, my code opens a website, looks for a specific text data and saves it in a csv document.
I thank you in advance for the help
You will have to check the terms of each cloud service as many do have downtime/restarts on their free tiers.
The kind of task you're describing shouldn't be very resource hungry, so you may be better off setting up your own server using a Raspberry Pi or similar.
I'll be setting up an webapp with Flask in an old Raspberrypi B+ running raspbian. The pi will also handle the desktop fuzz, so I'll try to keep it as light as possible.
The point of this question is mainly 1- what DB should I use? But I'm also wondering if 2- keeping it in a external usbstick would help? Let's take it step by step.
What DB: Consideration points
I rather do the programming using SQLAlchemy, so restrictions apply
The schema is not complex (around 10 tables)
Only one local user at first, probably forever, so a few querys and connections
Low overhead, the pi will most likely struggle, I'm just trying to minimize it.
The second point is about sd cards burnout. I read somewhere that any db should hit sd cards pretty hard and it got me thinking.
I'll set up some kind of external backup to this db anyway, but should I also keep the path to it in an stick? This should be really simple if I choose to use SQLite.
TYA
SQLite sounds like a perfect fit for this sort of use-case with embedded systems where you need a light-weight, yet full featured database. Many folks use SQLite databases on mobile devices as well for this reason: fairly limited cpu / memory resources, simple storage as a single file.
I have just coded a trading algorithm and some analytics software for the stock market which in itself works fine.
Since my computer is not always running or internetconnection is not running perfectly I would like to source the script out and put it on a Webserver for example, where it would run all day and night.
Do you guys now I could do that?
I would also like to build a user interface using django to monitor live performance.
Does anybody know what would be necessary to implement these steps?
Thanks in advance and kind regards
Marcel Kresse
This is very general question and the answer is close to "sky is the limit". As mentioned above, any cloud service provider will do.
Most (if not all) clouds have dedicated images for web servers and Django deployments. Have fun.
We have begun upgrading hardware and software to a 64-bit architecture using Apache with mod_jk and four Tomcat servers (the new hardware). We need to be able to test this equipment with a large number of simultaneous connections while still actually doing things in the app (logging in, etc.)
I currently am using Python with the Mechanize library to do this, but it's just not cutting it. Threading is not "real" in Python, and multiprocessing makes the local box work harder than the machines we are trying to test since it has to load so much into memory for Mechanize.
The bottom line is that I need something that will really hammer this thing's connections and hold a session to make sure that the sticky sessions are working in mod_jk. I need to be able to code it quickly, it needs to be lightweight, and being able to do true multithreading would be a perk. Other than that, I am open-minded.
Any input will be greatly appreciated. Thanks.
Open Source Testing Tools
Not knowing the full requirements makes it difficult, however something from the list might fit the bill.
In order to accomplish what I wanted to do, I just went back to basics. Mechanize is somewhat bulky, and there was a lot of bloat involved in the main functionality tests I had before. So I started with a clean slate and just used cookielib.CookieJar and urllib2 to build a linear test and then run them in a while 1 loop. This provided enough strain on the Apache system to see how it would react in the new environment, and for the record, it did VERY well.