Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

script to evaluate the trial iterations #298

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

MeWu-IDM
Copy link
Collaborator

@MeWu-IDM MeWu-IDM commented Dec 1, 2023

This script runs different iterations and compare its meanwith the deterministic calculations (no stochasticity)
model_inputs_iteration_check
this shows the trade off of run time and stability

@MeWu-IDM MeWu-IDM linked an issue Dec 1, 2023 that may be closed by this pull request
@BHagedorn-IDM
Copy link
Collaborator

This looks excellent. I think this is a good intermediate step to meet the immediate needs. Let's merge and keep discussing if we want to create something that is fully automated or not.

@BHagedorn-IDM
Copy link
Collaborator

Per conversation with Meikang and Rui, here are notes on two possible routes to implementing an automated system for selecting the number of runs, which would not be visible to the user and instead just run a scientifically-valid number of trials. TBD next steps in the new year. https://bmgf-my.sharepoint.com/:w:/g/personal/brittany_hagedorn_gatesfoundation_org/EV0DHld61gJAnuzZd3Yc2KIBmvtn9Ve1HjTc6gJ4n6zZow?e=bkRigJ

@rhanIDM
Copy link
Collaborator

rhanIDM commented Dec 13, 2023

https://kb.palisade.com/index.php?pg=kb.page&id=85#:~:text=To%20have%20%40RISK%20monitor%20convergence,for%20any%20or%20all%20outputs.
This website documents how At Risk (an add-in monte carlo simulation tool for Excel) tests convergence. It has a series of articles on specifying a convergence criteria and things to consider based on the test statistic we choose (mean, sd, or percentile). The approach appears to be similar to option 2 we discussed, and we could use these articles as reference to corroborate our calculations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

User request - appropriate choice of number of trials
3 participants