Develop Reference

Sort and order output data [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 months ago.
Improve this question
I have developed a small program that reads and outputs the live data of a machine. However, the data is outputted in a confusing and unordered way.
My question is, what exactly can I do to sort the output data e.g. in a table.
Best
enter image description here

You wrote many (topic, payload) tuples to a file, test_ViperData.txt.
Good.
Now, to view them in an ordered manner, just call /usr/bin/sort:
$ sort test_ViperData.txt
If you wish to do this entirely within python,
without e.g. creating a subprocess,
you might want to build up a long list of result tuples.
results = []
...
results.append((topic, payload))
...
print(sorted(results))
The blank-delimited file format you are using is OK, as far as it goes.
But you might prefer to use comma-delimited CSV format.
Then you could view the file within spreadsheet software,
or could manipulate it with the standard csv module
or various pandas tools.
When you review the text file next week,
you might find it more useful if
each record includes a timestamp:
import datetime as dt
...
results.append((topic, payload, dt.datetime.now()))

Is it better to have one large file or many smaller files for data storage? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 10 months ago.
Improve this question
I have an C++ game which sends a Python-SocketIO request to a server, which loads the requested JSON data into memory for reference, and then sends portions of it to the client as necessary. Most of the previous answers here detail that the server has to repeatedly search the database, when in this case, all of the data is stored in memory after the first time, and is released after the client disconnects.
I don't want to have a large influx of memory usage whenever a new client joins, however most of what I have seen points away from using small files (50-100kB absolute maximum), and instead use large files, which would cause the large memory usage I'm trying to avoid.
My question is this: would it still be beneficial to use one large file, or should I use the smaller files; both from an organization standpoint and from a performance one?

Is it better to have one large file or many smaller files for data storage?
Both can potentially be better. Each have their advantage and disadvantage. Which is better depends on the details of the use case. It's quite possible that best way may be something in between such as a few medium sized files.
Regarding performance, the most accurate way to verify what is best is to try out each of them and measure.

You should separate it into multiple files for less memory if you're only accessing small parts of it. For example, if you're only accessing let's say a player, then your folder structure would look like this:
players
- 0.json
- 1.json
other
- 0.json
Then you could write a function that just gets the player with a certain id (0, 1, etc.).
If you're planning on accessing all of the players, other objects, and more at once, then have the same folder structure and just concatenate the parts you need into one object in memory.

Web scraping a forum [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
My concern revolves around how to store the data I'm trying to retrieve data from certain threads of a forum. I want to be able to plot as much information as I want, so I don't want to store everything in a rigid structure; I want to be able to use as much info as I can (timezones more active, timezones more active per user, keywords throughout the years, points throughout posters, etc).
How should I store this? A tree with upper nodes being pages and lower as posts? How do I store that tree in a way it is easy* to read?
* easy as in encapsulated in a format I could export easily to other stuff.

I suggest to scrape only the posts (why would you ever need the pages?) into JSON, which you can keep in PostgreSQL in a jsonb field—it allows querying your JSON flexibly.
Later you’d write a script, or multiple, that would iterate over posts and do useful stuff like cleaning up the data, normalizing values, aggregating stats, etc.
See also
Someone wrote a post about PostgreSQL and querying JSON

Save data in Python and then read in C++ [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I have some code in Python which generates a set of data structures (they may be represented by classes but no methods are needed).
This data structures may be extended or added new structures in future
I have some code in C++ on Android which knew about part of this data structures and their fields.
The only way to pass data structures is through serialization to file and then deserialize them
Binary format support is needed.
Mature implementations in Python and C++ are needed. BSD, MIT, Apache licenses are preferred.
Speed is not critical.
I have tried custom format but it is hard to extend it.
SAX-like parsers are too low level for such task.

The most important factor here is the file format to be passed on - whether you need to create a proxy class on the other side or if you simply need to read data on the other side, once the data format is known on the received side, the receiver side should know how to handle it.
Thus, it is best to use the data format which are well-known and widely used. Mostly for the reason of their widely-used virtue, such data formats would also normally have some 3rd party or build in library to help you creating your data structure files.
For this purpose, I will recommend you to use either JSON or XML data format. Python already have serializers for both:
XML: http://code.activestate.com/recipes/577268-python-data-structure-to-xml-serialization/
JSON : https://docs.python.org/3/library/json.html, http://www.diveintopython3.net/serializing.html, https://docs.python.org/2/library/json.html
You can also search some of other alternatives which I believe are also available apart from them.

What is the best way to store data in Python? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
I'm creating a Skype bot for a group of friends, and I want to be able to have a somewhat login system for text-based games and storing information such as usernames, high scores, notes, friends list, messages, etc.
I was thinking of storing it in a text file named after the person's handle on Skype, however, I was wondering if there was a better way of doing it. Such as XML files.
I'd like to avoid SQL servers, and it's not like they're storing passwords so encryption won't be that much of a big deal. (I'd prefer local file storage. Something easily editable and delete-able)
I want to enable commands such as !note and !friends and !addfriend and !highscore and so on, but I need a method to save that information.

Have you considered pickle? It can store python objects (any object) to files so you can just load and use them.
import pickle
# Saving
data = [1,2,3,4]
pickle.dump(data, open("d:/temp/test.pkl","wb"))
# Loading
load = pickle.load(open("d:/temp/test.pkl","rb"))
For more info, read the docs
(Another option is the json module which serializes to json. It is used in a similar way, but can only save dictionaries, lists, strings and integers)