Open In Colab

Modeling Steps 5 - 10

Project guidance

Content creators: Marius ‘t Hart, Megan Peters, Paul Schrater, Jean Laurens, Gunnar Blohm

Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs

Objectives

This notebook contains the remaining steps of the how-to-model guide.

Implementing the model 5. Toolkit selection 6. Planning the model 7. Implementing the model

Model testing 8. Completing the model 9. Testing and evaluating the model

Publishing 10. Publishing models

Please see Steps 1 - 4 here first (or in the Modeling Steps section of the Project Booklet). We do not cover the two example projects in tabs here because you will want to dive into the code in the full project notebooks here and here (or in the Modeling Steps section of the Project Booklet).


Step 5: Selecting the toolkit

Video 6: Selecting the toolkit

Once you have completed Steps 1-4 to your satisfaction, you are now ready to model. You have a specific question, a goal in mind, and precise hypotheses expressed in mathematical language. All these components will empower you to chose an appropriate modeling approach.

In selecting the right toolkit, i.e. the right mathematics, computer science, engineering, or physics, etc approaches, you should consider the following important rules:

  1. Determine the level of abstraction

  2. Select the toolkit that best represents the ingredients and hypotheses

  3. Toolkit should express all needed relationships, mechanisms and concepts

  4. Keep it as simple as possible!

Guiding questions:

  • What is the most appropriate approach to answer your question?

    • What level of abstraction is needed?

    • Determine granularity / scale based on hypotheses & goals

    • Stay as high-level as possible, but be as detailed as needed!!!

  • Select the toolkit

    • Requires prior knowledge about flexibility / limitations of toolkit

    • Often more than one option possible

    • Some toolkits are more flexible, span a wider range of behaviour and/or are lumpable

    • Also determines how the model will be solved, i.e. simulated

      • Analytical? Numerical?

      • E.g. spatial, temporal resolution?

Viewing modeling as a decision process might help providing clarity regarding different model types, and how framing the problem and stating your goals influences the toolkit selection. Don’t be afraid to pursue goals that no one else pursues; diversity of models should be encouraged because it results in complementary model considerations.

Make sure to avoid the pitfalls!

Click here for a recap on pitfalls

Choosing a toolkit for the toolkit’s sake (e.g. DL because it’s cool to do deep learning)

  • this will prevent you to really answer your research question and/or speak to your hypotheses

Being at the wrong level of abstraction (see W1D1 intro)

  • too complex toolkits will have too many parameters you can't meaningfully constrain, and/or that add needless complexity
  • too simple toolkits will lack means to implement the details you care about

Not knowing any toolkits

  • this highlights a lack of literature review and/or background work to learn about the tools used by the field

Step 6: Planning / drafting the model

Video 7: Drafting the modeling

Planning the model involves thinking about the general outline of the model, its components and how they might fit together. You want to draw a model diagram, make some sketches and formalize necessary equations. This step will thus outline a plan of implementation. Once you have that plan, this will hugely facilitate the actual implementation of the model in computer code.

Your model will have:

  • inputs: the values the system has available - this can be broken down into data, and parameters

  • outputs: these are the predictions our model will make that you could portentially measure (e.g. in your idealized experiment)

  • model functions: A set of functions that perform the hypothesized computations.

You will thus need to define a set of functions that take your data and some parameters as input, can run your model, and output a prediction for a hypothetical measurment.

Guiding principles:

  • Keep it as simple as possible!

  • Don’t get lost in details

  • Draft on paper: start with a flow diagram

    • Draw out model components (boxes)

    • What influences what? (arrows)

  • Then consider each model box separately

    • Draft internal workings in terms of equations

    • This might require a lot of work…

    • Relate box inputs to box outputs!

    • Keep in mind that the model should include a way to relate model variables to measurements

    • Use the question, ingredients, and hypotheses to ensure you have all required components

Goal: Put in place all the components of the hypothesized relationships and explanations.

Make sure to avoid the pitfalls!

Click here for a recap on pitfalls

I don’t need to draft the model, I have it clearly in my mind

  • you might think you do, but experience shows you're likely missing many important aspects

I can just make a rough draft

  • the more detailed the draft, the easier it will be to implement the model in computer code
  • rough drafts tend to forget important details that you need to think about, e.g. signals needed (where do they come from?), parameters to specify (how to constrain them?), etc.

Draft is too detailed or not detailed enough

  • too detailed: you're writing our recursions, etc
  • not detailed enough: you have no idea what's inside "boxes"

Step 7: Implementing the model

Video 8: Implementing the modeling

This is the step where you finally start writing code! Separately implement each box, icon, or flow relationship identified in Step 6. Test each of those model components separately! (This is called a unit test). Unit testing ensures that each model components works are expected/planned.

Guiding principles:

  • Start with the easiest possible implementation

    • Test functionality of model after each step before adding new model components (unit tests)

    • Simple models can sometimes accomplish surprisingly much…

  • Add / remove different model elements

    • Gain insight into working principles

    • What’s crucial, what isn’t?

    • Every component of the model must be crucial!

  • Make use of tools to evaluate model behavior

    • E.g. graphical analysis, changing parameter sets, stability / equilibrium analyses, derive general solutions, asymptotes, periodic behaviour, etc.

Goal: Understand how each model component works in isolation and what the resulting overall model behavior is.

Make sure to avoid the pitfalls!

Click here for a recap on pitfalls

Building the whole model at once without testing components

  • you will make mistakes. Debug model components as you go!
  • debugging a complex model is close to impossible. Is it not woring because individual components are not working? Or do components not "play nice" together?

Not testing if individual components are important

  • It's easy to add *useless* components to a model. They will be distracting for you and for readers

Not testing model functionality step by step as you build up the model

  • You'd be surprised by what basic components often can alrealy achieve... E.g. our intuition is really bad when it comes to dynamical systems

Not using standard model testing tools

  • each field has developped specific mathematical tools to test model behaviors. You'll be expected to show such evaluations. Make use of them early on!

Step 8: Completing the model

Video 9: Completing the modeling

Determing what you’re done modeling is a hard question. Referring back to your original goals will be crucial. This is also where a precise question and specific hypotheses expressed in mathematical relationships come in handy.

Note: you can always keep improving our model, but at some point you need to decide that it is finished. Once you have a model that displays the properties of a system you are interested in, it should be possible to say something about your hypothesis and question. Keeping the model simple makes it easier to understand the phenomenon and answer the research question.

Guiding principles:

  • Determine a criterion

  • Refer to steps 1 (goals) and 4 (hypotheses)

    • Does the model answer the original question sufficiently?

    • Does the model satisfy your own evaluation criteria?

    • Does it speak to the hypotheses?

  • Can the model produce the parametric relationships hypothesized in step 4?

Make sure the model can speak to the hypothesis. Eliminate all the parameters that do not speak to the hypothesis.

Goal: Determine if you can answer your original research question and related hypotheses to your satisfaction. If the original goal has not been met you need to go back to the drawing board!

Make sure to avoid the pitfalls!

Click here for a recap on pitfalls

Forgetting to specify or use a criterion for model completion (in Step 1!)

  • This is crucial for you not to get lost in an endless loop of model improvements

Thinking the model can answer your question / hypotheses without checking

  • always check if all questions and hypotheses can be answered / tested
  • you will fail on your own benchmarks if you neglect this

You think you should further improve the model

  • This is only warranted if your model cannot answer your hypotheses / questions and/or meet your goals
  • remember: you can always improve a model, but you want to focus on the question / hypotheses / goals at hand!

Step 9: testing and evaluating the model

Video 10: Evaluating the modeling

Every models needs to be evaluated quantitatively. There are many ways to achieve that and not every model should be evaluated in the same way. Ultimately, model testing depends on what your goals are and what you want to get out of the model, e.g. qualitative vs quantitative fit to data.

Guiding principles:

  • By definition a model is always wrong!

    • Determine upfront what is “right enough” for you

  • Ensure the explicit interfacing with current or future data

    • model answers the questions/hypotheses/goals with a sufficient amount of detail

  • Quantitative evaluation methods (see also W1D3)

    • Statistics: how well does the model fit data?

    • Predictability: does the model make testable predictions?

    • Breadth: how general is the model?

  • Comparison against other models (BIC, AIC, etc.)

    • This is often not easy to do in a fair way… Be nice and respectful to other models.

  • Does the model explain previous data? (this is called the subsumption principle in physics!)

  • A good model should provide insight that could not have been gained or would have been hard to uncover without the model

  • Remember, a model is a working hypotheses; a good model should be falsifiable!

Goal: You want to demonstrate that your model works well. You also want to make sure you can interpret the model’s meaning and findings. I.e. what did the model allow you to learn that was not apparent from the data alone?

Make sure to avoid the pitfalls!

Click here for a recap on pitfalls

Thinking your model is bad

  • does it answer the question / hypotheses and meet your goals? Does it provide the leverl of explanation and insights you set out to gain? Then it's probably good enough!

Not providing any quantitative evaluation

  • Just do it, it's something that's expected

Not thinking deeply about your model and what you can learn from it

  • this is likely the most important pitfall. You want to learn as much as you can from a model, especially about aspects that you cannot gain from the data alone
  • model interpretation can open new avenues for research and experimentation. That's part of why we want to model in the first place! A model is a hypothesis!

Step 10: publishing the model

Video 11: Publishing the modeling

Guiding principles:

  • Know your target audience!

    • How much math details? How much explanation of math?

    • What’s the message?

    • Should be experimentalists in most cases!!!

      • Provide intuitive explanations, analogies, etc.

      • Famous researcher: “a good modeller knows how to relate to experimentalists”

  • Clearly describe what the goals, hypotheses and performance criteria were

    • Prevents from false expectation of what the model should be doing

  • A graphical representation is worth 1000 words (or more)

  • Show model simulations in parallel to data

    • Much more convincing!

  • Publish enough implementation details

    • A good model has to be reproducible!

Goal: Make sure your model is well received AND USED by the community. In order for our model to impact the field, it needs to be accepted by our peers, and order for that to happen it matters how the model is published.

Make sure to avoid the pitfalls!

Click here for a recap on pitfalls

not thinking of your target audience

  • no one will understand you if you write for yourself because only you know what you know...

forgetting about Steps 1-4

  • clearly spelling out steps 1-4 allows the reader to appreciate your goals and sets limits on model criticism
    • no one can tell you your model didn't do something you never set out to do
    • prevents unreasonable claims / rejection of paper

thinking you don’t need figures to explain your model

  • your model draft is a great starting point!
  • make figures that provide intuition about model behavior (just like you would create figures to provide intuition about expeimental data)

my code is too mesy to be published

  • not an option (many journal now rightfully require it)
  • code cleanly right from the start
    • model drafting should help with that
    • comment your code! Then comment more...

Writing the abstract of a modeling paper

Abstracts are very stereotyped pieces of writing that contain highly condensed information. To write a summary (= abstract) of your modeling, you can use these questions as a guide:

  • What is the phenomena? Here summarize the part of the phenomena which your modeling addresses.

  • What is the key scientific question?: Clearly articulate the question which your modeling tries to answer.

  • What was our hypothesis?: Explain the key relationships which we relied on to simulate the phenomena.

  • How did your modeling work? Give an overview of the model, it’s main components, and how the modeling works. ‘’Here we … ‘’

  • What did you find? Did the modeling work? Explain the key outcomes of your modeling evaluation.

  • What can you conclude? Conclude as much as you can with reference to the hypothesis, within the limits of the modeling.

  • What are the limitations and future directions? What is left to be learned? Briefly argue the plausibility of the approach and/or what you think is essential that may have been left out.

Instructions: write down your answer to each of those questions (1-2 sentences each, max!). When you’re done, stick the sentences together… Now you have an abstract!

Guidance for paper writing

There are good guidelines for structuring and writing an effective paper (e.g., Mensh & Kording, 2017), all of which apply to papers about models. There are some extra considerations when publishing a model. In general, you should explain each of the steps in the paper:

Introduction: Steps 1 & 2 (maybe 3)

Methods: Steps 3-7, 9

Results: Steps 8 & 9, going back to 1, 2 & 4

Here are some great materials to help you with paper writing:

The audience for all of this should be experimentalists, as they are the ones who can test predictions made by your model and collect new data. In this way your models can impact future experiments, and future data can then be modeled (see modeling process schematic below). Remember your audience - it is always hard to clearly convey the main points of your work to others, especially if your audience doesn’t necessarily create computational models themselves.

In addition, you should provide a visualization of the model, and upload the code implementing the model and the data it was trained and tested on to a repository (e.g. GitHub and OSF).

Suggestion

For every modeling project, a very good exercise is to first write a short, 100-word abstract of the project plan and expected impact. This forces focussing on the main points: describing the relevance, question, model, answer and what it all means very succinctly. This allows you to decide to do this project or not before you commit time writing code for no good purpose. Notice that this is really what we’ve walked you through carefully in this guide! 😀


Reading

Blohm G, Kording KP, Schrater PR (2020). A How-to-Model Guide for Neuroscience eNeuro, 7(1). https://doi.org/10.1523/ENEURO.0352-19.2019

Mensh B, Kording K (2017). Ten simple rules for structuring papers. PLOS Comput Biol 13(9): e1005619. https://doi.org/10.1371/journal.pcbi.1005619