Develop Reference

What to do with runtime generated data spanning several classes?

I'm a self-taught programmer and a lot of the problems I encounter come from a lack of formal education (and often also experience).
My question it the following: How to rationalize where you store the data a class or function creates? I'll make a simple example:
Case: I have a webshop (SHOP) with a REST api and a product provider (PROVIDER) also with a REST API. I determine the product, I send that data to PROVIDER who sends me back formatted data that can be read by SHOP to make a working product on the webshop. PROVIDER also has a secondary REST api that provides generated images.
What I would come up with:
I'd make three classes: ProductBase, Shop and Provider
ProductBase would be the class from where I instantiate and store the individual product information.
Shop would be where I design the api interactions with the webshop.
Provider same as shop, but for interactions with provider api.
My problem: At some point you're creating data that's not clearly separated in concern. For example: Would I store the generated product data (from PROVIDER) in the ProductBase instance I created? It feels like I'm coupling the two classes this way. But it not there, then where?
What if I create product images with PROVIDER and I upload them to SHOP? Do I store the uploaded image-url in PRODUCT? How do you keep track of all this info?
The question I want answered:
I've read a lot on OOP and Design Patterns, and I have adopted a TDD approach which has greatly helped to improve my code but I haven't found anything on how to approach the flow of at runtime generated data within software engineering.
What would be a good way to solve above problem(s) and could you explain your rationale for it?

If I understand correctly, I think your current concern is that you have "raw" product data, which you want to store in objects, and you have "processed" (formatted) product data, which you also want to store in objects. Your question being should you mix them.
Let me just first point out the other obvious option. Namely, having two product classes: RawProduct and ProcessedProduct. Which to do?
(Edit: also, to be sure, product data should not be stored in provider. The provide performs the action of formatting but the data is product data. Not provider data).
It depends. There are a couple of considerations:
1) In general, in OOP, the idea is to couple actions on data with the data. So if possible, you have some method in ProductBase like "format()", where format will send the object off to the API to get formatted, and store the result in an instance variable. You can then also have a method like "find_image", that goes and fetches the image url from the API and then stores that in a field. An object's data is meant to be dynamic. It is meant to be altered by object methods.
2) If you need version control (if you want the full history of the object's state to be available), then you can't override fields with new data. So either you need to store a history of every object field in the object, or you need to create new objects.
3) Is RAM a concern? I sometimes create dataclasses that store only the final part of an object's life so that I can fit more of the objects into memory.
Personally I often find myself creating "RawObject" and "ProcessedObject" classes, it's just easier a lot of the time. But that's probably because I mostly work with document processing, so it's very clear. Usually You'll just update the objects data.
A benefit of having one object with the full history is that it is much easier to debug. Because the raw data and the API result are in the same object. So you can very easily probe what went wrong. If you start splitting things up it's harder to track. In general, the more information an object has about where it's been, the easier it is to figure out what went wrong with it.
Remember also though, since this is a Python question, Python is multi-paridigm. And if you're writing pipeline-style architectures (synchronous, linear processes), then a functional approach can also work well.
Once your data is stored in a product object, anything can hold a reference to that. So a shop can reference an object and a product can reference the object. Be clear on the difference between "has-a" relationships and "is-a" relationships.

Same Kind Entities Groups

Let's take an example on which I run a blog that automatically updates its posts.
I would like to keep an entity of class(=model) BlogPost in two different "groups", one called "FutureBlogPosts" and one called "PastBlogPosts".
This is a reasonable division that will allow me to work with my blog posts efficiently (query them separately etc.).
Basically the problem is the "kind" of my model will always be "BlogPost". So how can I separate it into two different groups?
Here are the options I found so far:
Duplicating the same model class code twice (once FutureBlogPost class and once PastBlogPost class (so their kinds will be different)) -- seems quite ridiculous.
Putting them under different anchestors (FutureBlogPost, "SomeConstantValue", BlogPost, #id) but this method also has its implications (1 write per second?) and also the whole ancestor-child relationship doesn't seem fit here. (and why do I have to use "SomeConstantValue" if I choose that option?)
Using different namespaces -- seems too radical for such a simple separation
What is the right way to do it?

Well seems like I finally found the relevant article.
As I understand it, pulling all entities by a specific kind and pulling them by a specific property would make no difference, both will require the same type of work on the background.
(However, querying by a specific full-key, is still faster)
So basically adding a property named "Type" or any other property you want to use to split your specific entities into groups is just as useful as giving it a certain kind.
Read more here: https://developers.google.com/appengine/articles/storage_breakdown
As you see, both EntitiesByKind and EntitiesByProperty are nothing but index tables to the original key.
Finally, an answer.

Why not just put a boolean in your "BlogPost" Entity, 0 if it's past, 1 if it's future? will let you query them separately easily.

Django: how to transfer scattered data to a template?

Introduction
In Django, when the data you want to display on a template is included in one object, It's f**** easy. To sum up the steps (that everyone knows actually):
You Write the right method to get your object in your model class
You Call this method in your view, passing the result to the template
You Iterate on the result in the template with a for loop, to display your objects in a table, for example.
Now, let's take a more complex situation
Let's say that the data you want to display is widely spread over different objects of different classes. You need to call many methods to get these data.
Once you call these different methods, you got different variables (unsimilar objects, integers, list of strings, etc.)
Nevertheless, you still want to pass everything to a template and display a pretty table in the end.
The problem is:
If you're passing all the raw objects containing the data you need to your template, it is completely unorganised and you can't iterate on variables in a clean way to get what you need to display your table.
The question is:
How (which structure) and where (models? views?) should I organize my complex data before passing it to a template?
My idea on this (which can be totally wrong):
For each view that need "spread data" to pass to a template, I could create a method (like viewXXX_organize_data()) in views.py, that would take the raws objects and would return a data structure with organized data that would help me to display a table by iterating on it.
About the data structure to choose, I compared lists with dictionaries
dictionaries have key so it's cleaner to call {{dict.a-key-name}} rather than {{ tabl.3}} in the template.
lists can be sorted, so when you need to sort by date the elements you want to display, dictionary is not helpful, arghh, stuck again!
What do you think about all that? Thanks for reading until there, and sharing on this!

With your question you are entering in a conceptual/architectural domain rather than in a "this particular view of the data in my project is hard to represent in the template layer of django". So I will try to give you the birds view (when flying and not on the ground) of the problem and you can decide for yourself.
From the first philosophy box in the django template language documentation it's clearly stated that templates should have as little program logic as possible. This indicates that the representation of the data used in the template should be simple and totally adapted to the template you are trying to build (this is my interpretation of it). This approach indicates that you should have a layer responsible for intermediating the representation of your data (models or other sources) and the data that your template needs to achieve the final representation you want you users to see.
This layer can simple stay in your view, in viewXXX_organize_data, or in some other form respecting to a more complex/elaborated architecture (see DCI or Hexagonal).
In your case I would start by doing something like viewXXX_organize_data() where I would use the most appropriate data structures for the template you are trying to build, while keeping some independence from the way you obtain your data (through models other services etc).
You can even think of not using you model objects directly in the template and creating template specific objects to represent a certain view of the data.
Hope this helps you make a decision. It's not a concrete answer but will help you for sure make a decision and then staying coherent all trough your app.

How do to explicitly define the query used in subqueryload_all?

I'm using subqueryload/subqueryload_all pretty heavily, and I've run into the edge case where I tend to need to very explicitly define the query that is used during the subqueryload. For example I have a situation where I have posts and comments. My query looks something like this:
posts_q = db.query(Post).options(subqueryload(Post.comments))
As you can see, I'm loading each Post's comments. The problem is that I don't want all of the posts' comments, I need to also take into account a deleted field, and they need to be ordered by create time descending. The only way I have observed this being done, is by adding options to the relationship() declaration between posts and comments. I would prefer not to do this, b/c it means that that relationship cannot be reused everywhere after that, as I have other places in the app where those constraints may not apply.
What I would love to do, is explicitly define the query that subqueryload/subqueryload_all uses to load the posts' comments. I read about DisjointedEagerLoading here, and it looks like I could simply define a special function that takes in the base query, and a query to load the specified relationship. Is this a good route to take for this situation? Anyone ever run into this edge case before?

The answer is that you can define multiple relationships between Posts and Comments:
class Post(...):
active_comments = relationship(Comment,
primary_join=and_(Comment.post_id==Post.post_id, Comment.deleted=False),
order_by=Comment.created.desc())
Then you should be able to subqueryload by that relationship:
posts_q = db.query(Post).options(subqueryload(Post.active_comments))
You can still use the existing .comments relationship elsewhere.

I also had this problem and it took my some time to realize that this is an issue by design. When you say Post.comments then you refer to the relationship that says "these are all the comments of that post". However, now you want to filter them. If you'd now specify that condition somewhere on subqueryload then you are essentially loading only a subset of values into Post.comments. Thus, there will be values missing. Essentially you have a faulty representation of your data in the model.
The question here is how to approach this then, because you obviously need this value somewhere. The way I go is building the subquery myself and then specify special conditions there. That means you get two objects back: The list of posts and the list of comments. That is not a pretty solution, but at least it is not displaying data in a wrong way. If you were to access Post.comments for some reason, you can safely assume it contains all posts.
But there is room for improvement: You might want to have this attached to your class so you don't carry around two variables. The easy way might be to define a second relationship, e.g. published_comments which specifies extra parameters. You could then also control that no-one writes to it, e.g. with attribute events. In these events you could, instead of forbidding manipulation, handle how manipulation is allowed. The only problem might be when updates happen, e.g. when you add a comment to Post.comments then published_comments won't be updated automatically because they are not aware of each other. Again, I'd take events for this if this is a required feature (but with the above ugly solution you would not have that either).
As a last, hybrid, solution you could take the first approach and then just assign those values to your object, e.g. Post.deleted_comments = deleted_comments.
The thing to keep in mind here is that it is generally not a clever idea to manipulate the query the ORM makes as this could lead to problems later on. I have taken this approach and manipulated the queries (with contains_eager this is easily possible) but it has created problems on some points (while generally being functional) so I dropped that approach.

How to handle user mangement in Django for groups that has same access but different rules?

Background information:
I have created an internal site for a company. Most of the work has gone into making calculation tools that their sale persons can use to make offers for clients. Create pdf offers and contracts that can be downloaded, compare prices etc. All of this is working fine.
Now their sale persons have been divided into two groups.
One group is sale personal that is hired by the company.
The other group is persons a company themselves.
The question:
My challenge now is, that I in some cases need to display different things depending on the type of sales person. Some of the rules for the calculation tools will have different rules as to which numbers will be allowed etc. But a big part of the site will still be the same for both groups.
What I would like to know, is if there is a good way of handling this problem?
My own thoughts:
I thought about managing this by using the groups that is available in contrib.auth. That way I could keep a single code base, but would have to make rules a lot of different places. Rules for validating forms to check if the numbers entered is allowed, will depend on the group the user is in. Some things will have different names, or the workflow might be a bit different. Some tools will only be available to one of the groups. This seems like a quick solution here and now, but if the two groups will need to change more and more, it seems like this would quickly become hard to manage.
I also thought about making two different sites. The idea here was to create apps that both groups use, so I only would need to make the code for that 1 place. Then I could make the custom parts for each site and wouldn't need to check for the user in most templates and views. But I'm not sure if this is a good way to go about things. It will create a lot of extra work, and if the two groups can use a lot of the same code, this might not really be needed.
The biggest concern is that I don't really know how this evolve, so it could end up with the two groups being entire different or with only very few differences. What I would like to do, is write some code that can support both scenarios so I wont end up regretting my choice a half year from now.
So, how do you handle this case of user management. I'm looking for ideas techniques or reusable apps that address this problem, not a ready made solution.
Clarifications:
My issue is not pure presentation that can be done with templates, but also that certain calculation tools (a form that is filled out) will have different rules/validation applied to them, and in some cases the calculations done will also be different. So they might see the same form, but wont be allowed to enter the same numbers, and the same numbers might not give the same result.

you could use proxy models on the Group and User models that come packed with django.
then write your authorization and calculation methods inside the proxy model. if a new group is added later, you only need to add/change the methods inside of those two proxy models. then make every instance of Group and User (obviously only where necessary, not literally every one) find the proxy model instead of the actual contrib model.

If I'm understanding you correctly, it seems like you want to have two different groups have access to all the same views, but they will see different numbers. You can achieve this effect by making separate templates for the different groups, and then loading the appropriate template for each view depending on the group of the current user.
Similarly you can use a context processor to put the current group into the context for every view, and then put conditionals in the templates to select which numbers to show.
The other option is to have two separate sets of views for the two different groups. Then use decorators on the views to make sure the groups only go to the views that are for them.