Should You Put Butter in the Bag?

Intro

Should you put butter in the bag when cooking a steak sous vide? I would have never guessed this could be such a controversial question.

I posted a video to Instagram and Reddit that showed a steak being cooked sous vide with butter that received more angry comments than anything else I have ever shared. Some quick research made it clear that it was a controversial topic that hadn't been thoroughly tested.

I designed and conducted an experiment that reduced common cognitive biases around taste testing and flavor preferences and had a more statistically significant sample size than previous experiments.

The results are surprising.

Only interested in the results? Jump to the conclusion →

Background

After realizing how controversial the butter was, I researched the opinions of sources that I personally trust, as well as sources that were cited by commenters. Here is a list of those sources with the opinions that I was able to find.

1. Kenji Lopez-Alt said that he doesn't suggest adding fat (butter or olive oil) to the bag when cooking steak sous vide. He theorized that the added fat was absorbing fat-soluble flavor compounds from the meat and then being poured down the drain with the bag juice/butter. ¹

2. Chefsteps have suggested adding butter and olive oil to the bag with steaks in several of their recipes ²³ and guides ⁴.

3. Dave Arnold has previously suggested putting butter in the bag when cooking sous vide. ⁵

4. Modernist Cuisine adds suet (fat) to their sous vide steak recipe. ⁶

5. Sous Vide Everything published a video to YouTube where they did a blind taste test on a single person, who preferred the steak without the butter ⁷.

Experiment design

I designed an experiment—with the help of a few friends and the Modernist Cuisine book—to test two things:

1. Is it possible to differentiate between steak cooked with or without butter in the bag?

2. Did people who could tell a difference have a consistent preference?

The experiment was designed to reduce biases that commonly occur in preference tests and increase the sample size to increase statistical significance. These considerations make the findings of the experiment more trustworthy than previously completed experiments.¹⁷⁸

Modernist Cuisine suggests using what they call a ranking test, in which each test participant (taster) is presented with four samples, 2 cooked with butter and 2 cooked with no butter. The samples are labeled with a secret code, so the taster doesn't know which samples were cooked with/without butter. The taster must then rank the four samples in order of preference.

If butter (B) and not butter (NB) aren't consistently ranked first and second and second and third (B, B, NB, NB or NB, NB, B, B) we can conclude that the rankings are inconsistent and neither option is perceptibly better to that taster.

The preferences of the tasters who could consistently rank the samples would then show which option is preferred. In an attempt to maximize the statistical significance of the experiment, there were 5 tasters and each were given two sets of samples, each set of samples from a different steak and all of the samples in one set from the same steak.

There was one person, lets call them the investigator, that wasn't involved in the tasting. She was responsible for coding and randomizing the steaks so the tasters didn't know which samples were which and assuring that they were placed on the plate in randomized orders. Randomizing the order that the samples are tasted is important, as order can cause biases.

For each test, there are 24 possible ways to rank the four samples (4! = 4 * 3 * 2 * 1 = 24). There are 4 combination of rankings that prefer butter, 4 combinations of rankings that prefer no butter and 16 combinations of rankings that are inconsistent.

Based on these odds and an experiment with 12 tests,the expected values are:

8 Inconsistent (16/24 * 12)
2 Butter (4/24 * 12)
2 No Butter (4/24 * 12)

Experiment Execution

In order to reduce the number of variables and potential biases or inaccuracies, I chose a very simple preparation method. I started with 4 New York strip steaks, that were purchased together.

Each steak was then cut in half, optimizing for equal fat content. One half was sealed in a vacuum bag on its own and the other half in another bag, with one tablespoon of unsalted organic butter⁹. The steaks were cooked in the water bath for 2 hours at 56C/133F.

Two steaks (2 butter halves, 2 no butter halves) were taken out of the water, patted dry¹⁰ and seared at the same time in a very hot cast iron pan with canola oil.

The steaks were then transferred to a labeled cutting board so that the investigator could distribute them. She randomly placed 2 butter samples and 2 no butter samples on each plate, labeled with the letters A through D.

The plates were then presented to the tasters and they were asked to record their preference ranking for each sample.

After the first round of preferences were recorded, two more steaks were taken out of the water, seared, distributed and the steps to record each tasters preference were repeated.

At this point, we had enough steak for 2 more tests, so the investigator tallied the results and the 2 tasters with the least inconsistencies in their rankings (the tasters most likely to be able to differentiate between butter and no butter) did another round of samples.

Results

Of the 12 tests, 5 of them were inconsistent (the taster didn't consistently rank one type over the other), 6 preferred the butter variation and 1 preferred the no butter variation.

Below is a table of the results.

As shown in the Experiment Design section, the expected number of Butter preferences by purely guessing is 2. Our result of 6 is 3 times the expected value, making this outcome very unlikely to have happened by chance.

I don't know how to calculate the statistical significance of this, but if anyone else does, please email me riley@sousvide.blog, because I would love to include it.

Conclusion

1. Is it possible to differentiate between steak cooked with or without butter in the bag?

Yes! Seven of the twelve tests were able to tell a difference, but overall, all tasters agreed the presence of butter in the sous vide bag had a small impact on flavor.

2. Did people who could tell a difference have a consistent preference?

Yes, they preferred the steak cooked with butter in the bag. Of the seven consistent rankings, six of them preferred butter and only one preferred no butter.

Considering the above, I generally recommend that you should add butter to the bag when cooking steak sous vide.

Does this mean you always should be putting butter in the bag when you cook steak sous vide? No. Taste preferences are unique to each person and can change over time. There are also many other fats, like olive oil, to consider using instead of butter.

If you are cooking for yourself or others that you have done these tests with and you prefer the steak cooked without butter in the bag, you should probably keep cooking it that way.

If you haven't performed an experiment to determine your personal preference or are cooking for people you can't run the experiment on, you should probably be adding butter to the bag.

Want to know the next time I publish something? Follow Sous Vide Diaries on Instagram or signup for my email newsletter.

Notes

• I've reached out to the sources referenced in the introduction and will amend the post with any responses that I receive.

• The results of these experiments should only be considered when cooking steak sous vide. It's more widely agreed upon that putting fats in the bag when cooking fish is a good idea. I might do a follow up study for this.

• Thank you to the tasters: Andrew, Madelena, Nick and Steph. Thank you to Andrew, Rhett and Steph for proofreading.