I want to develop a Math quiz program using reinforcement learning.
Assume that we have 1000 questions in hand and 25 questions to be asked in each quiz. Instead of asking questions at random, program has to learn from the way user answer and ask the next question.
Quiz programme should be a reinforcement learning agent. How to design the solution and which are the reinforcement learning techniques to be used?
Example :
BoT: what is 5+ 1:
User: 3 (Wrong Answer)
Bot: Asked easy question or if correct answer asked a difficult question.
PPO is a very common technique for these type of RL applications in the edTech space. You can take a lot of inspiration from this article. They use the RLgraph package and the PPO algorithm.
You would first have to define your goal/reward function. In your case, I would define the reward function to have something to do with the percentage of previous questions that was answered correctly. If this percentage is 0% or 100%, the reward is low (it's too difficult/easy). If it's close to 50% you might choose the reward to be high.
That way, the algo will move towards questions that get 50% correctness (are medium difficulty). You can play with the range (last 2 q's or last 10 q's).
As state-space, you can also include questions answered correctly, maybe characteristics like age etc. to help initiate the algo when the user hasn't used it too much yet.
As action-space, you can have all the questions. You could also cluster(e.g. difficult/easy, or geometry/algebra) questions based on your intuition and make the clusters actions, to decrease the action space.
Related
The quick summary of my question:
I'm trying to solve a clone of the Flappy Bird game found on the internet with the Reinforcement Learning algorithm Proximal Policy Optimization. Apparently, I've faced an issue with designing the reward system. How can I specify a reward for the agent given that it's a third party game so it does not return anything to me and the only info I get is visual information form the window?
Some details and the background:
Prior to trying to solve a third party game I've played with several google gym environments such as Cart-Pole, Mountain Car, Lunar Lander and recently Car Racing. To solve them I used PG, DQN, Actor-Critic and PPO algorithms. After understanding how to work with problems when the state is an image I've decided to take on a new challenge and try to get out of the sandbox (gym).
I've picked Flappy Bird because it's simple in concept, action space is 1 (actually 2) and it's notoriously hard for humans.
My code can be found here: https://github.com/Mike-Kom/Flappy-Bird-PPO
Agent class and buffer was tested on the Car Racing so there shouldn't be any issues with the RL algorithm. The neural net was changed a little due to a different state size but conceptually it's the same so there should not be any problems ether.
My current guess is that the reward system is not robust and causes the agent not to learn properly.
Currently I'm just giving the agent 0,025 points each step and 2 points after the 25th frame and above (I've found that this is exactly the frame at which the agent passes between the first two pipes.) but it does not seems to work.
Any suggestions on how to solve an external environment and especially on how to design the reward system are welcome!
Sorry if the code is messy and not professional it was originally meant to be just for me :) Programing is just my hobby and my occupation is far from code writing.
Moreover, this is my first question here and I wanted to take an opportunity and thank all of you for writing our answers and suggestions for different question! You make this community super helpful for so many people! Even though, I did not write a question before I found here a tone of answers and good suggestions :)
The goal is that the user enters a number between 1-10 and AI tries to figure out what number the player picked.
For example I take the number: 6 and AI then tries to find out which number with 3 attempts.
AI learns which numbers are frequently chosen and tries them.
The problem is. I'm new to machine learning and thought this would be a good project to learn about machine learning. I can't find such a project anywhere and I don't know where to start. I've already asked this question on other platforms but it looks like nobody can help me.
I thought I use python, keras and tensorflow for that
Has anyone done this before or know where it is?
Thanks :)
It looks like you assume your user gives non-uniform distributed data.
However, this problem may not really related to ML.
One of the easy solutions is to let users choose, again and again, you collect the data. You will likely find a distribution that fits your data (if your user really does so). Then you can use that distribution to predict.
Take a look at scipy.stats.rv_continuous.fit
I realize that this is slightly outside the realm of what sort of questions are normally asked here, so please forgive that. I have been tasked with an open ended technical screening for a job as a data scientist. This is my first job that has asked for something like this, so I want to make sure that I am submitting really good work. I was given a dataset and asked to identify the problem and how to use machine learning to solve it, give stats on the target feature, pre-process the data data, model the data, and interpret the results.
I am looking for feedback about if I am missing anything huge in my results. High level feedback is fine. Hopefully some of you are data scientists and have either had to complete a technical screening like this or have had to review one and can offer some valuable feedback to an up-and-coming data scientist.
Thank you!
Github Link to Project
have a look on the
Mars Express Power Challenge Get the data, model and predict the
thermal power consumption
here https://kelvins.esa.int/mars-express-power-challenge/
The chalenge was to get the data and predict future consumption of the orbiter to plan how to save energy (when in the solar field there is a risk of over heating, and in the solar night a risk from being to cold)
The teams used different approach LSTM is probably the one I would choose. But the winning team conducted a very detailed explanation on the "Feature Engineering and Selection".The point is what is important is not the tool used but the correct choice of feature extraction and selection.
https://arc.aiaa.org/doi/pdf/10.2514/6.2018-2561
I read both the winning paper and your work. Really I prefer your way.
As you see if you read the paper, your methodology is quite comparable, but they put the feature extraction study at the center of the research.
You may secure your work by providing more evidences that you picked the right method for the FE. For exemple you could provide 2 method of FE and compare the result given the method, or, you explain you chosen one knowing the current state of the art about this particular paper which prove blablabla...
You could add the comparative result of ARIMA VAR VARMA and yours to illustrate the "outperform" and reference on papers of the state of the art for the past 3 years on the field, and other references on recent publication on LSTM for energy consumption prediction.
Your document end abruptly one would wait for a decorative conclusion as we are used to find in a regular paper.
That it.
(please dont take account of my only opinion as I don't feel myself data-scientist :) I will be very proud of myself the day I would be abble to produce what you done ;) thanks for sharing it was nice to read it)
If I was the evaluator, I would ask questions like,
1) What is the research/business problem?
Suggestion: Begin the report by clearly specifying the question
2) What are the existing solutions to solve the problem?
Suggestion: Add a brief literature review on existing solutions for similar problems and their results preferably in a tabular format.
3) Briefly elaborate on the descriptive and multivariate properties of the data.
Suggestion: Add descriptive and inferential statistics on the data including some preliminary hypothesis that can be derived from the variable correlations.
4) Why did you choose this particular approach to solve the problem?
Suggestion: Give credible justification backed up by quantitative hypothetical example solutions, that are in favour of the proposed approach.
5) If it's a classification task, I would ask a question like, "What is the baseline accuracy of the model?" And if its a clustering task, "What is the baseline for cluster purity?"
Suggestion: Find this accuracy from the target variable distribution.
Finally, you need to understand, why such an open-ended question is asked. There can be two possibilities;
(a) The company is new with reference to data science and is unsure of what they are looking for, meaning, they do not have either the required expertise to evaluate the candidate skills or they are simply unsure of what is their requirement. If this is the case, then it's imperative that the report is as simple and detailed as possible. Stay away from throwing jargon.
OR
(b) the company is experienced in data science and this is a filtering test. To filter out the self-proclaimed data scientist nincompoops, who think chaining some ready-made solution steps (like preprocessing, dimensionality reduction, modelling) solves a problem. The underlying idea is to figure out the analytical capabilities of a candidate.
Therefore, write the report wisely and ensure nothing is falsified.
Best of luck.
It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.
Closed 10 years ago.
So I've been thinking of creating a program similar to this in Python. The problem is that I know next to nothing about AI. Right now I'm reading up on genetic algorithms and neural networks and such, but there's a lot of things to absorb.
My largest problem now is I don't know how the entire thing is put together. So if anyone can tell me the general framework for a program like this, I'll be really grateful. For example, how the creatures can "see" things in their surroundings, the kind of "chromosomes" they have and how the environment is created.
Also, can anyone suggest suitable libraries for AIs, any engines, etc? I've been thinking of using neurolab, will that be a good idea?
Thanks!
Interestingly, I've created what you're describing, in Python.
Here's a pic:
The green dots are food, and the red ones are organisms that need to evolve and learn how to eat the food blocks.
The best source of inspiration and guidance for something like this is definitely Polyworld: youtube.com/watch?v=_m97_kL4ox0.
Another great example which helped me learn about backpropagation, which I suggest you should learn too, is here: arctrix.com/nas/python/bpnn.py.
Although this question is simply too vague to answer correctly, the general tip is that you first get a solid ground on how ANN's function, and then construct a program with this general algorithm:
Check for user input (should the camera move? should the program quit?)
For each organism:
Check if it is alive, etc.
Take input from the world
Evaluate its neural network
Apply the results to the physical body
Apply positive or negative feedback (as you'll see, it all comes down to this)
Update the world (you can use a physics engine)
Optionally, draw the world
For example, your input neurons could correspond with physical or visual stimulation. Is there something in front of me? Then set neuron X to 1, otherwise to 0.
And your output neurons could map into forces. Is neuron Y active? If so, apply an impulse to this organism's in this world cycle.
I also recommend that you don't use other people's libraries for your neural network computation, because this is a great way to actually learn how to implement them yourself. You could you Pygame for rendering, and PyBox2D for physics, though.
Now, about "seeing" other organisms... These problems are best solved when isolated. You need a good software design, that will enable you to divide the whole problem into many subproblems, which are easier to solve.
In general, seeing can be done by raycasting. Create a directional, normalized vector (where the length is 1 unit, that is, sqrt(x*x + y*y) == 1), and then scale it (x *= factor; y *= factor;). That way, you get a line that's stretching outwards. Before you scale it, you can then loop through all of the organisms in the world (you'll have to optimize this), and check if the vector is inside of them. If it is, calculate its length, and your organism has some information about its surroundings.
There are many simpler ways. For example, divide space around each organism into four quadrants, which four represent eyes (X - up, right, down, left of the organism). Then calculate the length from the organism to every other organism (optimize!). Find the closest organism. Find out in which quadrant it is. Voila, an active eye!
I hope I was helpful, but you'll have to be more precise and come back when you have an actual problem. Read the FAQ as well.
http://scikit-learn.org/ is a great resource (and framework) for you to learn about machine learning in python.
If you want to build a neural network, http://pybrain.org/ is a good framework.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I am a machine learning beginner. I'd like to learn the basics by teaching computers to play checkers. Actually, the games I want to learn are Domineering and Hex. My language of choice is Python
These games are pretty easy to store and the rules are much simpler than chess, but there aren't too many people who play. If I can get this idea off the ground it would be great for experimenting Combinatorial Game Theory to see if a computer and find the optimal move.
I found this old paper on checkers from the 1960's by a guy at IBM. Originally I had asked about neural networks, but they are saying it's the wrong tool.
EDIT: It could be that machine learning is not the right strategy. In that case, what goes wrong? and what is a better way?
You might want to take a look at the following: Chinook, Upper Confidence Trees, Reinforcement Learning, and Alpha-Beta pruning. I personally like to combine Alpha-Beta Pruning and Upper Confidence Trees (UCT) for perfect information games where each player has less than 10 reasonable moves. You can use Temporal Difference Learning to create a position evaluation function. Game AI is probably the most fun way to learn machine learning.
For links to all of these topics, click on
http://artent.net/blog/2012/09/26/checkers-and-machine-learning/
(I was not able to include more links because the stack overflow software considers me a newbie!)
Get the book called "Machine learning" by McGraw Hill and read the first chapter. It's extremely well written and the first chapter will teach you enough to make a program that plays checkers. Personally I made a program that plays 5 in a row on miniclip.com, also in python.
http://www.amazon.com/Learning-McGraw-Hill-International-Editions-Computer/dp/0071154671
When playing checkers, you seek to gain an advantage over your opponent by taking his or her pieces and crowning your own. Losing your pieces and allowing your opponent to crown his or her pieces is not desirable, so you avoid doing it.
Board game engines usually revolve around a position evaluation function. For checkers, my first guess would be something like this:
score = number of allies - number of opponents
+ 3 * number of crowned allies - 3 * number of crowned opponents
Given a board, this function will return the score of the board. The higher the score, the better your position. The lower the score, the worse your position.
To make a naive checkers "engine", all you need to do is find the best move given a board position, which is just searching through all immediate legal moves and finding the one that maximizes your score.
Your engine won't think ahead more than one move, but it will be able to play against you somewhat.
The next step would to give your engine the ability to plan ahead, which essentially is predicting your opponent's responses. To do that, just find your opponent's best move (here comes recursion) and subtract it from your score.