I am trying to solve an issue for my app, which is programmed in ReactNative.
I have a type of object named 'Card' which has two main variables, both of them strings: 'question' and 'answer'.
The data for this objects is provided to the app, which has to load them and show them to the user. I have around 10 thousand "points" of data written as lines in a .txt, each line being: "this is a question? // here is the answer".
I would like to create a Python script that breaks that .txt in 10 thousand files that can be read by a ReactNative method at run-time. I thought about AsyncStorage.getItem and wanted to ask how to format these files so that they can be read by the method.
But I'm beggining to think I would need to use Expo.FileSystem.readAsStringAsync(). Is that right? I would rather use AsyncStorage.getItem and just parse the file right away...
AsyncStorage is not the thing you're looking for. It's a key-value-storage, which doesn't solve your problem as you're trying to use an already existing text file as database so to say. Expo's built in solution Expo.FileSystem.readAsStringAsync() seems to do exactly what you want. There are probably more ways to solve your problem, but using Expo's API is a pretty good solution it'd say.
If you want to improve performance I'd suggest using an actual database since you won't be needing to parse the txt file to the appropriate format. You could use an SQLite Database and just populate the data on the app's first launch similar to this approach How can I embed an SQLite database into an application?. Or you could call a remote api, but that's probably out of your scope.
Related
I am developing a web application in which users can upload excel files. I know I can use the OPENROWSET function to read data from excel into a SQL Server but I am refraining from doing so because this function requires a file path.
It seems kind of indirect as I am uploading a file to a directory and then telling SQL Server go look in that directory for the file instead of just giving SQL Server the file.
The other option would be to read the Excel file into a pandas dataframe and then use the to_sql function but pandas read_excel function is quite slow and the other method I am sure would be faster.
Which of these two methods is "correct" when handling file uploads from a web application?
If the first method is not frowned upon or "incorrect", then I am almost certain it is faster and will use that. I just want an experienced developers thoughts or opinions. The webapp's backend is Python and flask.
If I am understanding your question correctly, you are trying to load the contents of an xls(s) file into a SQLServer database. This is actually not trivial to do, as depending on what is in the Excel file you might want to have one table, or more probably multiple tables based on the data. So I would step back for a bit and ask three questions:
What is the data I need to save and how should that data be structured in my SQL tables. Forget about excel at this point -- maybe just examine the first row of data and see how you need to save it.
How do I get the file into my web application? For example, when the user uploads a file you would want to use a POST form and send the file data to your server and your server to save that file (for example, either on S3, or in a /tmp folder, or into memory for temporary processing).
Now that you know what your input is (the xls(x) file and its location) and how you need to save your data (the sql schema), now it's time to decide what the best tool for the job is. Pandas is probably not going to be a good tool, unless you literally just want to load the file and dump it as-is with minimal (if any) changes to a single table. At this point I would suggest using something like xlrd if only xls files, or openpyxl for xls and xlsx files. This way you can shape your data any way you want. For example, if the user enters in malformed dates; empty cells (should they default to something?); mismatched types, etc.
In other words, the task you're describing is not trivial at all. It will take quite a bit of planning and designing, and then quite a good deal of python code once you have your design decided. Feel free to ask more questions here for more specific questions if you need to (for example, how to capture the POST data in a file update or whatever you need help with).
I developed a software which can be automatically updated, so I need external-placed config file/files. For now I use json file to store user-input variables like user name etc. But I am not sure how the program itself should be controlled. I mean things like checking if program is opened for first time after update to know if update notes should be shown, what functions were already used etc. For now I am doing it with things like:
if os.path.exists(control_file_1):
actions_1
if os.path.exists(control_file_2):
some other actions unrelated to actions_1
it is independent from the files content - so there is no need to read the file content - which is convenient.
What functions should be used to store those information in one file and read them efficiently? Just normal file.read() etc? It seems not very clean-code friendly.
Thanks
UPDATE:
Looks like a ConfigParser is a way to go. Am I right? Or are they any better ways to accomplish what I am going for?
Given that you need to have config information stored in a file. If you choose to have that information in a file that contains a json record then it is the most convenient if the file is used internally and updating and reading the record in the file is easy (treat it as a dict)
However, if you want a more universal config.ini reader then you can go with ConfigParser class which you can use directly or create your own wrapper
class MYConfig_Parser(ConfigParser):
so that you can check stuff in the constructor like if mandatory entries are available etc before processing the entries.
I have an original text that I want to translate. I normally do it manually but I know I could save a lot of time translating automatically the most frequent words and expressions.
I will find out how to translate simple words, the problem is not here. I have read some books on python and I think using string manipulations can be done.
But I am lost about how to create the output file.
The output file will contain:
short empty forms ready to be filled wherever there is text that has not been translated
the translated words wherever they were in the original file
In the output file I will fill manually the empty forms, after pressing Tab the cursor should jump to the next exmpty form
I am lost here, I know how to do forms on html but the language I am used to is Python.
I would like to know what modules from Python I could use. I need some guidance on this.
Can you recommend me a book or a tool that explains how to do something similar to this?
This is what I want to do, assuming I have managed to create a simple database to translate colors from Spanish to English.
The first step contains the original file.
The second step contains the automatic translation.
In the third step I complete the manual translation.
After finishing everything is grouped into a normal txt file ready to be used.
I think it is quite clear. I don't expect people to tell me the code to do this, I just need to know what tools could be used to achieve my goal.
Thanks for editing.
To create an interface that works with a web browser, Flask for Python is a good method for creating webforms. There are tutorials available.
One method for storing data would be an SQLite file. That may be more than you need, so I'd recommend starting with a CSV file. Libraries exist in Python for both CSVs and SQLite.
I want to make a python program (with a PyQt GUI, but I don't know whether that is relevant) that has to save some information that I want to store even when the program closes. Example for information I want to store:
The user can search for a file in a file dialog window. I want to start the file dialog window in the previously used directory, even if the program is closed in between file searches.
The user can enter their own categories to sort items, building up on some of my predefined categories. These new categories should be available the next time the program starts.
Now I'm wondering what the proper way to store such information is. Should I use pickle? A proper database (I know a tiny bit of sqlite3, but would have to read up on that)? A simple text file that I parse myself? One thing for data like in example 1., another for data like in example 2.?
Also, whatever way to store it I use, where would I put that file?
I'm asking in the context that I might want to later make my program available to others as a standalone application (using py2app, py2exe or PyInstaller).
Right now I'm just saving a pickle file in the directory that my .py file is in, like this answer reconmends, but the answer also specifically mentions:
for a personal project it might be enough.
(emphasis mine)
Is using pickle also the "proper, professional" way, if I want to make the program available to other people as a standalone application?
Choice depends on your approach to data you store, which is yours?:
user should be able to alter it without usage of my program
user should be prevented from altering it with program other than my program
If first you might consider deploying JSON open-standard file format, for which Python has ready library called json. In effect you get text (which you can save to file) which is human-readable and can be edited in text editor. Also there exist JSON file viewers and editors which made viewing/editing of JSON files easier.
I think SQLite3 is the better solution in this case as Moldovan commented.
There is a problem in pickle, sometimes pickling format can be change across python versions and there are greater advantages of using sqlite3.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I'm working on the project. I need to create a website. On it, my colleagues will be able to upload files with data in Excel format on the server. This will be a table with standardized column names. These files need to be attached to the database. The data is stored in SQL format. I will need to process this data and show some statistics on them. I plan to implement these transformations on the python. As a result, i will need to display certain graphics. All this will be done on the server of the company.
I didn't have to solve such tasks before. Advise me, please, how can I convert new files from Excel to sql on the server? And how to automate this? I could not find anything about it. And also what language is best used to create a site so that it can work with the python and the sql?
I'm not sure how to automate it more clean and idomatic for SQL-Server (maybe it has some built-ins to handle excel files), but maybe this thoughts may help you as a more 'general' approach:
1) Since you want python, the best web-framework for the site would be Flask or Django. It depends on what functionality you would like to add to this site later. Current task looks like Flask is ok for it, since it is smaller and simplier than Django.
2) There are a lot of python libraries to work with excel for python. Here is a question and answers about parsing excel files with python: parsing excel documents with python I would say that it makes sense to create some web form where user uploads excel-file, and you placing it on the servers disk (and it makes sense to rename that file somehow to avoid overriting files with the same names) into predefined directory.
3) Next step - to read that file from disk with one of tools for parsing excel files and get needed columns from it. Maybe you would also want to reformat that data somehow.
4) Final step - create a pure SQL query with python (there are a lot of libraries for it too, I guess) with the data you got from file to INSERT it into some table. If you do not want to work with pure SQL queries, you can take some ORM to work with it. Like SQLAlchemy. But it looks like overhead for this task for me.
Here also a documentation on how to work with file uploads with Flask: http://flask.pocoo.org/docs/0.12/patterns/fileuploads/