Can two different modeling programs be used to confirm the results of each other?

Question

Computer Models

Computer modeling is used in various engineering fields. I am specifically considering structural analysis or finite element analysis (FEA). Sometimes models are used to speed up repetitive calculations that could be done by hand. Sometimes models are used to perform calculations that are not easy or even possible to do by hand.

Checking

There are a few standard methods for checking the results of computer models.

• Run verification models and confirm that the results match a previously calculated answer.
• Run simple models that can be checked by hand calculations.
• Test physical models.

The problem with the first two checking methods above is that they either only check specific situations or they only check the simple parts of the program.

The physical model method can be expensive for full size models and scale models may not always give the same results as the full size.

This leaves a gap in what results can be checked. For any complicated model, there is no easy way to check that the results from the program are correct. The engineer must trust that the software produced correct results from the model.

Comparison Check

The model could be input into two different programs (made by different companies). The assumption is that if the results of the two models are similar enough, then the results should be correct for the model used. This would not catch any errors in creating the original model, but it would catch errors in the software implementation.

• Could two separate programs be used to check the "correctness" of the results from the model?
• Would using this method of comparing a model in two separate programs provide the same level of assurance in the results as any of the other checking methods?
• What could be the disadvantages of using this procedure?

Answer 1

Yes, getting a second opinion can be useful. This is done routinely in weather forecasting where exact solutions are unknown, and there is some judgement about how to apply various factors.

There will be less wiggle room in something like a finite element mesh stress analysis because the iterative equations for solving it will be basically the same no matter who wrote the software. The real issue is not solving the mesh as much as creating a good enough mesh in the first place.

One way therefore to get multiple opinions is to vary the mesh parameters. Hopefully you still get pretty much the same answer. If you make the mesh 2x finer and get a significantly different answer, then that's a strong clue the original mesh wasn't detailed enough. You also then don't know for sure the next mesh is detailed enough without making a even more detailed one and getting about the same answer.

Of course nowadays mesh generation itself is somewhat automated and adaptive. This is no longer about just the physics of solving the mesh, but include heuristics about when and how to subdivide. Different software can vary in this regard, so running two different programs with the same initial data can be useful.

Answer 2

I write this from the perspective of an engineer who develops simulation software.

I think the practice described is bad, and I recommend you do not use two different softwares to "confirm" the results.

In general, two different modeling softwares can not be used to confirm much anything other than their similarity. Two softwares could easily both get two similar but wrong answers, especially if they use similar models. I can think of at least one instance where this is definitely the case, and Trevor Archibald mentions another. I would be more impressed by two softwares which use different modeling techniques getting similar results.

This subject is called the verification and validation of computer models, and it has a fairly vast literature. I'll offer a sketch of the basics. Verification is comparing a model against an "exact" solution (which could be a hand calculation, or something more complex), that is, checking the math of the software. The assumptions behind the exact solution could be wrong, but at the very least you'd want to make sure the software gets the math part right. Validation is comparing a model against an experiment. This allows you to check whether the model you are using is accurate, which is something verification can't do for you.

The problem with the first two checking methods above is that they either only check specific situations or they only check the simple parts of the program.

This is a real problem facing software developers and users. There are established ways to handle it that are much better than comparing two different softwares.

The issue is that you can never test every possible case. Your software might pass case A, but case A doesn't involve physics X, Y, or Z, and that makes you totally off case B. So, what you'd want are a large number of checks that cover at least all of the basic features you want to check. Many softwares have "V&V suites" which are basically exactly that.

In terms of verification, there are numerous options. You could generate new exact solutions for different cases. Sometimes this alone is adequate. However, as you've noticed, oftentimes what you can do by hand is limited to very simple cases. For the more general cases, you could use a technique called the method of manufactured solutions (Google it). This requires programming and can get messy, but allows you to test basically anything you could think of. (The messiness part can be handled via a library like MASA, by the way.)

Also, I'd like to point out that contrary to what Olin Lathrop suggests, with the method of manufactured solutions, you can generate what are, for the purpose of testing, exact solutions. They are not "exact" in the strict sense, because they do not exactly satisfy the equations the software is solving without modification. But they satisfy equations which are very close, and the difference is accounted for to make the test rigorous. This technique is not very popular at the moment, but it has been used to test things which previously were thought to be difficult to test.

In terms of validation, you could look for more experimental data or run your own experiments.

Answer 3

I think this is a good practice overall.

By using two different softwares, you may be able two avoid two kind of errors: 1) errors that come from an inaccurate software (which should not be overlooked), 2) errors that come from the lack of habit of the user with the software (hidden options, default settings...).

If the softwares are different enough, the odds of getting two times the same wrong results are low.

However, the errors that come from a bad modeling choice cannot be avoided this way. So I'd say the main disadvantage is to over-trust the results because they have been confirmed by two softwares.

I do think that it is better to master one software, running all kind of test cases (comparison with academic results for instance), than using several softwares and having only an average knowledge. Moreover, I think it is best to know the basics of FEM analysis, and using only two "one-click-to-run" softwares is a bad practice because users are likely to reproduce modeling errors.

PS: I am writing as an electromagnetism/thermal FEM analysis user (no other domains).

Answer 4

Answer from the point of view of a Design Engineer

Checking the results of one program against another will give you some level of certainty that the results are correct. It is unlikely to give you 100% certainty, but then that level of certainty is hard to achieve.

A big issue I see is being able to transfer the model from one piece of software to another. Although model import/export is being improved by most software companies (because of BIM), I wouldn't expect all features of a model to be exportable. Geometry is relatively easy to import/export as the exchange file only needs to contain coordinates. But e.g. member end releases are likely to be stored very differently by different software, so unless/until a universal file exchange format is agreed, I suspect a lot of effort would be required to completely rebuild a model in the second software program.

Based on my own experience, errors in results are far more likely to come from incorrect input of data or incorrect assumptions than they are from poorly written software. The time and effort in using independent software to verify an answer is therefore probably not a good use of your time.

Answer from the point of view of a Software Engineer

Verifying software against other software isn't taken as justification that your software is correct. It is far better to find published data/results that can be used to verify that software gives correct answers. Imagine a sales meeting where a software company is trying to sell their software to an engineering company:

Engineer: How do we know that your software is correct?

Software salesman: Well, we checked it against our competitor's software and got the same answer.

Engineer: So you're saying that your competitor is sufficiently better than you that his software is the yardstick against which you measure your software? Sounds like we should buy his software instead!

Answer 5

I concur with the other answers here, that this in general can be a good idea and will help ensure the accuracy of simulation results. In terms of how good it is in relation to the other verification methods, I'd say previously known results and physical tests are both better options if feasible, but hand calculations may require over-simplification if the model is sufficiently complex.

What I really want to point out is something that hasn't been addressed on the final point: potential weaknesses of this practice. Using two different FEA packages may catch a peculiarity of one package that causes an error, provided you can identify which analysis is correct and which one is off. However, there are some general limitations to FEA overall, regardless of method or implementation. Sharp corners and other stress concentrators that cause singularities in the model aren't something that will change much from package to package, these will always be weak spots. This is where engineering knowledge and intuition is required.

I've done simulations on parts I know stand up easily to certain stresses, and the model shows that the internal stress is 10x the yield strength; this is obviously incorrect, because it's on an involute spline pattern and the FEA software doesn't like that.

Lastly, it should be obvious that changing software doesn't eliminate user error. If you make an error in the model or parameters, that error will screw you up no matter what you use to analyze it.

Answer 6

The boundary conditions and modeling technique will greatly influence the results. What I suggest is running a simplified/idealized version (like planar, or axisymmetric) and a full solid one and comparing the two.

The problem with two different FEA software is that under the hood the solvers are going to be largely the same. The differences observed would be from different convergence criteria maybe, or from different assumptions on how boundary conditions are applied. You are not checking the model, but the ability of each solver to know it has reached a solution.

I think FEA results should be validated by first common sense and hand calculations, then by similar but different models (solids instead of beams for example) and finally by physical testing to see if parts fail where and how the FEA predicts. The when a part will fail is more difficult because it is infuenced by manufacturing processes, material variations and resudial stresses.