An Introduction to Practical and Applied Winery Laboratory Quality

- [Beth] The extension specialist at Penn State and we're here as part of the Eastern Viticulture and Enology Extension Forum.

Just letting you all know that the presentation is being recorded and will be published online for future access, but just so you're aware of that.

And this is part of a multi-part series.

We have Chris Gerling also as our counterpart at Cornell University.

There are many viticulture webinars.

I believe the next one, viticulture one, is maybe next February the 28th.

I'm not quite sure. Check it out.

- Well, the next one actually is, we're gonna skip a month and then we'll have it again in April.

April 18th will be the Pet-Nat round table.

And that link will be coming, that'll be coming soon.

So watch for it.

- Yeah, we're gonna do a round table with people from York State, Pennsylvania, and most likely Virginia, talking about Pet-Nat production and the rise in interest in that.

So today our agenda is that Dr. Patricia Howe will be presenting.

And then we will have time for questions both during the presentation and afterwards we'll try, we'll be wrapping up around 4:30.

Just one edit, please use the Q&A tool not to chat tool.

And Molly Kelly will be monitoring that.

So, I'm going to go ahead and exit this so that Pat can get her presentation pulled up as I introduce her.

Okay. So let me stop sharing.

There we go.

And all right, there we go.

And sorry you couldn't see my face before.

All right, so today, we are delighted to introduce Dr. Patricia Howe as our guest speaker.

Dr. Howe has over 35 years of production winemaking experience and enological problem solving, having worked at or consulted for dozens of wineries across the United States.

She has a BS in Fermentation Science and an MS in Food Science with a sensory science emphasis from UC Davis and a PhD from Cornell University, where she contributed to our more nuanced understanding of sulfur dioxide chemistry in wine.

Her business experience includes operating 16019, once the smallest freestanding bonded winery in the US, opening Ascent Laboratories LLC and applied sensory and analytical wine lab in Napa, California, and starting a satellite branch of ETS labs in Oregon.

Dr. Howe has taught wine making, wine analysis and winery sanitation, including to yours truly at Cornell University, Napa Valley College, and through the Vesta program and managed the UC Davis teaching labs.

Pat is also past president of ASEV and was co-founder of the California Enological Research Association, which I did not know before this presentation actually.

She currently serves as co-chair of the ASEV ad hoc Lab Proficiency Committee.

She's a frequent presenter and writer on wine chemistry, sensory and quality topics.

Dr. Howe is currently employed by UC Davis as a lecturer in the Department of Viticulture and Enology, and as an instructor in their continuing education program.

Welcome, Pat.

- Boy, I feel really old after all of that.

Oh, okay.

Well hi, everybody.

How are things in Pennsylvania and the East Coast?

I've been hearing about the weather there and reminiscing of my time at Cornell but not, - I think you had a much harder winter than this one.

By the way, Pat, you're still in, switch the presenter, we're seeing your presenter mode.

So switch screen. - Thank you.

That was a problem before, wasn't it?

So let's, give me a second here to, I'm gonna have to do this.

I'm gonna have to do this.

Huh.

Are we better?

- Yes. Beautiful. - Okay.

- All right, thanks.

- Yeah, I was telling Beth and Molly before that, I think I need three screens instead of two when we do these presentations because of the multiple areas you have to manage.

But anyway, I'm glad to be here.

We're gonna try and cover some basics of some practical and applied winery laboratory quality issues.

But because of the diversity of the group that's probably on the call today and the differences in experiences and ways that we should be addressing some of our questions, what I really wanted to do to start off with is do a series of polls, asking some questions about your personal experiences in the lab and trying to find out from you how things work.

So what we have is to start with, excuse me, I'm getting a prank call right now.

There that's gone a series of polls for me to get to know you a little bit better, and then we'll have some discussions about the results of those polls.

We're then going to go through what Gavin Sachs calls a thought experiment.

So we're going to play a little game of trying to think about a certain situation and how it's gonna apply.

And this is gonna be specifically designed talking about titratable acidity.

And I'm hoping that most of you are able to run titratable acidity in your labs or you've run them before so that you can participate in this little thought experiment.

And then we're gonna go through how to interpret the results of that experiment.

And then finally, we're gonna end up talking a little bit about some applied tools that you can use in your laboratory to track and manage the actual variation that you have in your analytical testing.

So hopefully we'll be able to get through all of that in the time allotted.

All right.

So,, we're gonna now try to launch a whole bunch of these polls.

- I just hit launch, Pat. - All right. I can see them.

So these are multiple answers.

Choose as many as apply in your situation for each of these questions so that we'll get a kind of idea of what's going on out there.

So there is five questions, so go ahead and work through them.

And as we work through those, we can take a look at the results when they get through.

So, I'm seeing nobody's participating yet on my screen.

- Yeah, likewise.

Unless it's that, does it log in? - It might be that they don't show up until all of them are chosen.

Maybe that's it.

- Yeah, hopefully.

Oh, there's one person. - IS that okay?

- Okay. Yep.

Three.

Yeah. Okay.

We're on a roll. Okay.

- [Pat] I was holding my breath there, Beth.

I was like, what did I do wrong?

- Me too, But not you just you.

Just ignore.

- So, someone says they can't submit their poll.

- Let's see.

If it won't let you submit, did you do all five questions?

- [Molly] Yeah, there should be a submit.

Sam, maybe just go down a little bit more on your screen and maybe you're just not seeing the submit button.

Yeah, you have to scroll down to all five questions.

Frank chimed in.

Yeah, so Drew says, "You have to scroll down, answer all five questions, then hit Submit." And Bruce, I see where you raised your hand.

If you could type your question into the Q and A.

You think we have everybody?

Wanna end the, - It says we have 36 of 48, so I don't know whether the other folks are not at a place where they can participate or if they're having trouble with the submit button.

- Yeah, I can go ahead and end it, Pat.

- Yeah, we can end it. - And I mean, we're pretty good.

Yep, yep. Okay.

Yep.

- We have enough, - Yeah, enough participation.

Yep. 75% participation. Not bad.

So you can see the shared results.

- Yes, okay. All right.

So, can everyone see the shared results?

Does that work?

Everyone can see them. Is that correct?

- I'm not getting anything in the chat.

Can anyone let us know if you're able to see the results of the poll.

- Whole bunch just came through?

Yes, yep. Yes, yes - All right.

Okay. - Yep, yep.

- All right.

Thanks for doing that, you guys, and for your patience with the polling function with a large group like this.

So the reason I'm asking this question is to understand the group better.

And the first question for you was, who's actually doing lab analysis in your winery?

And I can see that a lot of you are doing them yourself and that it's a motley assortment.

Everyone from the winemaker, enologist, if you have a chemist, lab technician, cellar master, cellar workers, in some cases everyone does it, 17% of you actually send these out and then there's somebody who actually does something different.

So at some point, whoever the person is that says other, I'd be curious if you could go ahead and put that in the Q and A what other means for who performs the lab analysis?

Did I miss a category there?

The reason I'm asking this is because I need to know if I'm talking to the right people.

And because all of you have done this, I think we're gonna get good participation.

So the fact that 81% of you actually do this analysis yourself will help the discussion quite a bit.

All right.

My next question for you is a really basic question of, how do you know when you run an analysis if it's done correctly?

And this is sort of the crux of what we're gonna be talking about today.

And I put these answers in here because these are the answers I've heard over the course of my professional career.

And some people say, I know it's right because I did it myself.

And having been a lab technician for a long time, even though I do it myself, there are things that can go wrong that I would never have imagined of.

So although that is an answer that I see a lot, I want you to think about what you mean when you say, I know it's right because I did it myself.

I think what you're saying is that, you have controlled for every error that you know about as opposed to having to trust the performance of somebody else.

The next answer is that we follow the instructions.

And by following the instructions, you feel that you are probably gonna get the right answer.

And I'm just gonna use as an example of trying to follow a recipe from a recipe book, that you can follow the recipe and still have the cake come out flat or something go wrong.

So just by the fact that following the instructions is a good start, it doesn't mean those instructions are gonna work with your equipment and your ingredients.

So, that's the other thing just to be aware of.

So, following the instructions and doing it yourself will cover a lot of that problem, but it's not the be all and end all.

The same with taking a class.

It's sort of like following the instructions and doing it yourself that supports all the errors that could go wrong but not all of them.

We talk a lot about the reasonableness of your results and that is usually something that says, oh, I got a number that doesn't make any sense at all.

Something must have gone wrong.

Or contrarily, I got a number that seems reasonable, therefore it's right.

And the problem with this is that, if you're only getting reasonable results, then why are you even testing?

The reason that we test frequently is to find the results that are not normal.

So this is part of the catch 2 2 of lab analysis is that if you only get good answers, then why are you running them?

And if you never get wrong answers, then why are you running them?

So there's a little problem with this sort of concept of saying that it's a reasonable number, therefore I did it right.

Comparing to a previous result is sort of the same issue is, you're testing to see if there's something going on and if you compare to a previous result and it's the same number, then why are you testing?

So this starts to become this complex self-fulfilling prophecy of what the results are that I'm getting.

Having worked for one of the outside labs that people frequently will compare their numbers with, this is a problem that causes a lot of anxiety because when you compare results with a third party, there is an implication in that there's only one right result.

And the problem is, is that there is no such thing as one right result.

There's a range of right results.

And we're gonna talk about that with the rest of the presentation.

There's a range of results.

And that if you send to an outside lab and the result that you get from that one analysis is on the lower end of that range, you're always gonna be comparing to that lower number.

So, the challenge that we're gonna have is, unless you send it to an outside lab enough times to understand what that distribution is, there's not a lot of value in just comparing to an an outside lab.

Now we get to someone, this is where we really want to go.

And that's running a control check sample and why that has the power over all of these other methods.

So, we'll get to that, but that's something I'd like to see in the future when we ask this question, we need to see higher than 36%, 'cause that is really gonna give you the most input and understanding of whether your result is right.

It's not the be all and all again, but it is probably one of our best choices.

All right.

And then I did wanna know how you guys are running TAs because that's gonna be our thought experiment today.

And I see that you're using color endpoints and pH endpoints and auto titraters.

So we're looking for the most part at people doing a titration based TA.

And because it is called a titratable acidity, that's good.

There are other methods.

You can use IR scans and that's a secondary method and a few other things.

But looks like our number of people doing those are quite low.

So that will facilitate our discussion today. That's good.

All right.

And to make sure I'm speaking the same language, if I talk about gram's per liter as tartaric in a TA, most of you will know what I'm talking about.

Those of you that are doing percentage or grams per 100 mls, my values that I say are gonna sound 10 times higher than what you're used to.

So just keep that in mind.

If you're used to reporting a TA of 0.72%, I'm gonna be speaking in 7.2 grams per liter.

So just adjust your expectations of what a normal number is.

And finally, the number of digits that we talk about.

One of the hardest things I teach in class is the concept of the old school kind of significant figures thing that they had you memorize all these stupid rules and no one could ever quite figure out how it worked.

A lot of those significant digit exercises are based on in the old days when we had slide rules, You know, slide rules, you only had so many digits you can work with and you had to decide which ones to do.

We don't have slide rules anymore.

We have calculators and phones.

So, I like to speak about instead of significant figures, I talk about the number of digits that you report because that gives you the degree of precision that will help understand diversity in your answers.

So some of you apparently are only talking about a TA of 7 versus a TA of 8.

Some of you would be talking about a TA of 7.1 versus 7.2.

Some of you 19% actually do a 7.24 versus a 7.30 or something.

You're reporting a total of three digits.

And a few of you don't know or inconsistent.

But most of you are looking at that two digit and then equal amounts at one or three.

And I'll go over why that's important, okay.

So that was our, we can probably stop the sharing now and close the poll.

All right. So, that's good.

Thank you for letting me get to know you a little bit.

All right.

So now we all know about each other and how we do titratable acidity.

We're gonna do an exercise now using titratable acidity as an example.

And the point of this exercise is to understand inherent variability in analysis.

I spoke a little bit about that.

When you send out samples for analysis, there's not one right answer, there's a range of answers.

And that's what we're gonna try and get to a better understanding of what that's all about.

So once we get comfortable with that concept of variability, then we can talk about how to monitor and manage it, okay.

So we're gonna do this thought experiment.

Here's our thought experiment.

Imagine running a titratable acidity every day for 30 days and whether or not you would get the same result, okay.

I want you to think about this.

We're gonna open a poll with the whole series of possible results.

But I want you to think about what you think your results would be and then choose the poll response that's closest to what you concluded, okay.

So think about, this is your thought experiment, I'm gonna run a TA every day for a month.

My wine itself is not changing, okay.

The wine is not gonna be any source of variability.

It's gonna be anything else associated with running that test, okay.

So think about that and then choose the response that's closest to what you kind of imagined, okay.

And that poll is launched now, okay.

- [Molly] And so Pat, while they're taking the poll, I got a question in the chat.

And someone asked, compare results from previous doesn't mean that it has to be the same at all.

- Okay. - You need to clarify that.

- Okay. So it's a relative question.

- [Molly] Maybe we could ask Edward, could you clarify your question just a little bit?

What question would you have Pat, for him to clarify?

Yeah, he just said, "Wait, compare results from previous doesn't mean that it has to be the same at all." - Yeah, okay.

But I think my point would be that it's still a question of what your expectation is and whether comparing it to a previous is gonna help you feel that it's accurate.

Whether it's changing a direction you think it's gonna be or not changing a direction you think it's gonna be.

- [Molly] If that doesn't answer your question, Edward, just please type again in the Q and A.

Thank you. Thanks, Pat.

- Okay, we have about half, yeah, this wine is completely stable.

It's not changing at all.

Any variation you get in this test has nothing to do with the wine.

We are somehow magically getting a wine that will be the same every single day.

Yeah, you should be able to choose multiple, oh, it says single choice, okay.

Yeah. Well, pick this closest one.

Pick the closest one.

- Sorry, Pat. - It's not a problem.

I think this is one I did say choose the one that's closest.

- I think so. - Yeah.

Okay, I'm showing 75%. - Great. Yeah, yeah.

- Okay. Yeah, go ahead.

- [Beth] All right.

We're gonna go ahead and end this and share results.

Here we go.

- Thanks, you guys, for doing this.

I'm hoping when you thought about this, you had some questions about what was going on.

The fact that we did say it, it's not gonna be the wine that changes.

So we did get one person say, they get exactly the same number every time.

We got a few people saying, if they got a different result, it's because somebody else ran it or they did something wrong.

And then we have a bunch of you saying, you know, there's gonna be a range, but you don't know what it is.

And a few of you saying, I think it's gonna be within a gram per liter, between a half a gram per liter, or between a 10th of a gram per liter.

So there's some expectations here of what a normal variability would be with titratable acidity.

But there's no consensus with the group.

So when you think about it, here's a test that we run frequently all the time, fairly simple test, but even in a group of 70 people, we don't know what our variability is.

And that to me is very revealing.

So in fact, I think Molly, if you wanna talk about, if you were in a hospital group and you talked about, what's the variability of the blood sugar number, there would be a much better understanding of what that is.

- [Molly] Correct?

- Yeah, so let's talk about this and how we would get to understand, for our own facility and for our industry, what is a normal expected variability on something like TA?

All right.

So first question is, is we would not expect to get exactly the same result.

And why is that?

Is because there's variability in our analysis that has nothing to do with the sample itself.

And we're gonna go through for a traditionally run titratable acidity, what the source of some of those variations are.

And even if people are running this test perfectly, making no errors whatsoever, there is going to be a variation that's exclusive of the errors that could happen from not following the instructions or not paying attention or some things along those lines.

So we're gonna go through a case of absolutely perfectly run analysis, you know.

There's not any mistakes being made and look at what the variability is, just in the analysis itself based on some of the challenges that we have just in running analysis, okay.

So these are perfectly run.

I'm not gonna talk about somebody made a mistake or they're using the wrong whatever.

This is, if you run them perfectly, what kind of variability might we see and where would it come from?

Okay.

So first we're gonna have to talk about, what is a titratable acidity.

And if you look at our definition of titratable acidity, and because so many of you're doing it with a titration either by pH endpoint or color endpoint, we know that we're looking at calculating titratable acidity using the milliliters of sodium hydroxide times the normality or the molarity, interchangeable in this case of our sodium hydroxide, divided by the milliliters of the wine sample.

And then there's a conversion factor to get that from equivalence to grams as tartaric acid.

So, if we were gonna run this perfectly, those are the three sources of error.

I'm gonna take away that 75 'cause the 75 is really not a problem.

But if we were gonna look at the variation, it's gonna become from the mls of our sodium hydroxide, the normality of our sodium hydroxide and the mls of our wine sample.

We're gonna remove that 75 because that's not gonna be a source of error.

Typical methods that I see, and for the purpose of this thought experiment, we're gonna use these values.

Typical analytical methods are going to use 0.1 normal sodium hydroxide and 5 mls of the wine sample.

So if your method's different, the concepts are still gonna be the same.

And I'm gonna say this is gonna be a typical analysis because I need to have a concept of what a typical amount of sodium hydroxide would be 'cause these are the values that we're gonna use to try and calculate the impact of all the errors that we're gonna have.

So for this thought experiment, we're gonna say we're looking at 0.1 normal sodium hydroxide.

We're gonna use 5 mls of our wine sample.

And on the average for a TA of about 7 1/2 as a sort of a typical TA for a white wine maybe, we're gonna say that we'd use 5 mls of sodium hydroxide.

We're only using these values because we're gonna look at the impact, the errors are gonna have on a percentage basis to these values.

So let's look at our first source of variation.

We're looking at the mls of sodium hydroxide.

So what could impact that?

Well, the first thing that could impact it is gonna be the buret or if you're using an auto titrator, you have a automated buret.

If you're doing it manually, you're gonna have either a glass buret or one of these, a piston style burets.

And there's errors in these burets.

So the quality of the buret we'll dictate how much error you're gonna have in just delivering the mls of sodium hydroxide.

So there's class A glass burets, the error in them is 0.05 mls.

Over the course of the 50 mls of the buret.

The piston buret has a 0.05 error over 5 mls.

Class B glassware has twice the variability of tolerance of class A.

And this is all assuming, because we're gonna do perfect analysis, that this is a 20 degrees centigrade.

If you're running in a particularly warm or cold facility, your volumes and differences might be higher.

But we're doing perfect analysis under perfect conditions.

And that means that just the choice of your buret alone is that you're gonna have between a 0.1% error and a 1% error just from the volume of your delivered sodium hydroxide.

So what kind of buret are you guys using?

We're gonna have one more poll here.

I wanted to just kind of hear from you guys if you know, if you have a class A or a class B or if you're using a piston or what?

If you're doing auto titrator?

If you don't know, that would be another.

Okay. This is pretty interesting.

So I think Beth and Molly, when we get up to, to that 70% number, that's probably gonna be a good time for us to go ahead and stop the polls.

That seems to be about the participation level that we're getting.

- [Beth] Sounds good. And three, two, one.

The poll is closing.

- (laughing) All right.

So this is great, you guys.

35% of you know, you have a class A glass buret.

So that means you know that your error is probably gonna be about 0.1% on the volume that you have for your mls.

Those of you at class B, you're gonna have twice that.

If you don't know, you might wanna go back and look.

You can tell just looking, I don't know if you can see my mouse, but if you look at your buret, it should have a big giant A out there if it says a Class A and that should help you out a little bit.

All right. So, that is one source of our error, right?

We know that the buret is one source.

And even though we're gonna talk about the perfectly run analysis, I wanna point something else out that there is a human error factor in here that can be significant, and that's the accurate reading of your volume.

And this is something that in my class at Davis, we spent a lot of time on because reading a buret and reading meniscus is a new skill for a lot of our students.

So there is gonna be a human error in reading, which we're gonna remove because we're doing the perfect analysis for this thought experiment.

But there's also a meniscus issue and a rounding error that depends a little bit on the method that you're using and how many digits you record when you transfer or write down or enter into your formula the mls of sodium hydroxide.

There's a whole nother thing here about the ability to correctly fill burets that we're not gonna go into because in this experiment, thought experiment, it's a perfectly done one.

So what we have here is a picture of a glass Class A buret with a volume.

The picture on the left and the picture on the right are actually exactly the same.

The difference is on the right.

Somebody's using what's known as a buret reading card that just shows a black card behind the buret.

Enables a better reflection of the bottom of the meniscus.

So I want to want you to take a look at this buret and tell me what value you would record for that buret reading.

So formulate your response, look at that, and then pick the answer that's closest to what you would write down if you were calculating a TA on that value.

And I can tell already that you guys do this more often than my students do because I have this question on a quiz and they do very poorly.

I think we're getting pretty close.

- [Molly] Yep.

- All right. And three, two, one, The poll is closing.

- All right.

So in theory, when you are looking at a buret reading like this, you're supposed to estimate that third digit.

So in classic chemistry, the correct answer here would be 1.42 mls.

But in all the laboratories I've ever worked in that do wine analysis, I'm actually pretty impressed that 35% of you would actually do 1.42.

That's higher than I've seen in actual production laboratories.

1.4 is the value that most of the places I know would be using.

If you have anything besides 1.4 at 1.42, you might wanna look more closely at that meniscus scale and understand where you would read a meniscus on a buret.

All right.

So that is just a side question to a certain extent because we're not gonna be looking at the human error associated with this.

So, we're not gonna consider the meniscus error, except it can be pretty high.

So, if you look at what the meniscus error could be, it could be 2%, right?

So that's something that we're not gonna consider for the rest of the exercise.

But I think it's important to call out.

That is probably a pretty common source of variation.

All right.

So that is how to look at the variation in a perfectly run test from just the mls, the sodium hydroxide that we use.

If we're looking instead at the normality of the sodium hydroxide, if people are purchasing sodium hydroxide, that's at 0.1 molar or 0.1 normal.

If you look at the price that you're paying and what the manufacturer will guarantee as the acceptable range of molarity or normality that you're gonna see, the highest quality supplier that I can find states on their label that if you purchase 0.1 molar, it will be between 0.0995 and 1.1005 mols per liter.

Now this is interesting because when we purchase sodium hydroxide for class, we buy the cheapest stuff we can find.

And it turns out that the values that we have for our class goes from 0.09 to 0.110 mols per liter.

That is a pretty big variation just by what you're paying.

If you're actually doing in-house manufacturer and you're making this from sodium hydroxide pellet, so you're diluting a higher concentration, there is gonna be quite a bit of error associated there that I have no way of easily quantifying.

So, for the purpose of a thought experiment, we're gonna look at sort of the commercially best practice available concentrations versus sort of the cheapest version.

And that can range from 0.5 to 10% variation.

In-house, there's no way for me to calculate that.

And it's gonna depend on all of your standardization methods.

Then that's beyond the scope of this particular presentation.

But for the thought exercise, we're gonna say it can range from 0.5 to 10%.

So the question now is, what's the variation of the sodium hydroxide that you use for your TA?

And do you know what the range is?

Does it say on the label, not just 0.1, but 0.1 plus or minus whatever normality that you're getting?

And depending on where you're buying this, it might not tell you.

Some of the local winemaking supply stores will just say 0.1.

So, if you know what your range is and you can remember what it is, you go go ahead and enter that information into the Q and A and we'll get a sense from those of you that are out there if you know whether you're at that commercial best or commercial not so good level.

- [Beth] All right.

And we're, oh, we just jumped again, We're gonna close in five, four, three, two, one.

End poll.

- Okay, this is encouraging.

Half of you say that you know what the variability is because it says on the label.

The rest of you may not know how much variation you have in your sodium hydroxide.

If you calibrate in-house, again, that's gonna be beyond the scope of this presentation because we'd have to look at a lot of the other sources of variation.

But I think we could probably say for our thought experiment that we can use the the best versus the worst as sort of a working range for our error.

All right.

The last source of the variability for our thought experiment is delivering the volume of our wine sample that we're gonna titrate.

And typically, people are gonna be using pipettes of some type.

And just like with the burets, there's gonna be Class A volumetric pipettes that will have the best performance of 0.012 mls of error in, of course, of the 5 mls delivered.

Class B will be twice that.

And again, this is at 20 degrees.

Volumetric pipettes are very typical shape.

They've got this bulbous part that contains the main part of the volume, and then straight sides above and below.

But there's also something called a measuring pipette, which is the straight sided pipette.

That also comes in class A and class B, but the errors are quite a bit higher than the volumetric pipette.

And then there's these mechanical pipettes, like a pipette man or a epiendorf, they have much higher error, and that's even when they're properly calibrated and maintained.

And that's just part of the whole ASTM standards for glass burets, for piston pipettes, that's just inherent in the mechanical nature.

Despite their convenience, they're not as precise as some of the glass pipettes.

So, aside from the accurate delivery issue, aside from the improper use of pipettes, assuming a perfectly run pipette, you're gonna have an error with a Class A pipette of 0.24% all the way to 0.8% or 0.9% variability with a mechanical pipette or a Class B measuring pipette.

So again, another large source of error.

And again, I wanna ask you, what kind of pipette are you using to deliver the volume of wine sample into your titratable acidity?

- [Molly] So, going back to the sodium hydroxide question that you asked, Pat, Drew says that he uses commercial best sodium hydroxide.

He says it's very expensive from Metrohm.

- Metrohm, okay.

So he's probably using auto titrator.

Drew, if you know what the range is on that sodium hydroxide, if that would be kind of curious to know.

- [Molly] Someone also put into the chat that they use a syringe to measure their wine sample.

- Wow, okay.

So that's gonna be even higher error than what we have, unless it's some kind of syringe I don't know about.

So we're gonna say, if you're thinking about that, when we start looking at our best versus worse scenarios, if you're using a syringe, is this like a GC kind?

A glass barreled syringe?

- [Molly] Or maybe just one of those plastic, you know.

- It depends.

They do have those like, needle syringe things that can be pretty precise.

It's what we had before we had mechanical pipettes.

- [Beth] They have really low volume though, if it was a GC syringe, but maybe.

- They have larger ones. I just don't know.

Okay, we're at 64. - 64, yeah.

So five, four, three, two, one.

End poll.

- All right.

So almost half of you are using volumetric Class A pipettes.

That's good. That's the about as best as you can do.

Those of you that can afford a mechanical pipette, the speed and convenience sometimes outweighs the I precision issue because they're pretty handy.

If you're using measuring pipette straight sided, if you're using a pipette, just go ahead and upgrade to the volumetric.

It might slow you down a little bit, but you'll get a little extra position if you're interested in that.

But let's look at our overall variation now that we've looked at all of the sources that we have.

Let's do a couple of scenarios.

Let's say we are using the best equipment that we can use.

We're gonna use a Class A buret, we're gonna get the best commercial sodium hydroxide we can get, and we're gonna use a Class A pipette to deliver our wine sample.

If we add up all of these variabilities, we're gonna basically say that the variation that we're gonna get over the course of this month's analysis where we run the same sample every day, is that we're gonna see if our actual true value is seven and a half grams per liter.

We're gonna have a plus or minus 0.063 gram per liter variability.

So those of you that said that you thought that if you ran this every day, you would see something like a 0.1 variability.

You're basically telling me that you run this test perfectly all the time and you're using this equipment.

Otherwise you're not gonna be able to get anything that good.

And let's look at the other case.

Let's say you're using a Class B buret.

let's say there's a bigger factor with the rounding error from your buret.

Let's say you're using not the best commercial sodium hydroxide in the world, and you're using a mechanical pipette.

So instead of now having an error of plus or minus 0.84%, you're looking at an error of plus or minus 12.1%, which means that the same line run every day for a month, that this lab up here is getting an error of plus or minus 0.063.

You're gonna get plus or minus 0.9075.

So, the variability that you would get over the course of that month would range from a 6.59 to an 8.4 gram per liter on the same wine of 7.5 TA, okay.

And these are all analyses that are run perfectly.

So you can see the variability that you're gonna get is beyond your control to a certain extent.

And that it is just inherent in the case of running this test every day.

There's going to be variation.

You can't get exactly the same result every single time if you do it 30 times, because there's gonna be variation here you can't control.

- [Molly] So Pat, here's some updates on some of the chat.

So the syringe is plastic, probably poor quality.

Anne said that it depends on the sample.

She said, "If the sample's chunky juice, a measuring pipette might be used.

Clean wines, I use a measuring pipette.

- Okay.

There are options.

It's gonna be hard to find these days because it's all like three manufacturers worldwide now.

But they used to have a volumetric Class A buret that had a wide tip that like a grape seed could get through.

And those were really valuable on juices and new wines.

But I don't know how hard they are to get now.

But really, is this an acceptable variation, is the next question, right?

So do you need to have this degree of precision in something like a TA or, I mean, is this an acceptable variation?

How identical do you need your TAs to be?

I mean, no one's gonna die if you get something off by 0.1 grams per liter.

And 0.1 grams per liter versus 1 gram per liter is a big difference.

So let's try and figure out now the other question, which was, how many digits do you use to report your results?

So if you are only reporting your results using one one digit, like 7 grams per liter versus 8 grams per liter, if you're only using that one digit, you're gonna not see this level of variation no matter what tools you're using.

You're losing that resolution that's gonna allow you to see that variability.

So you're hiding your variability by not reporting more digits.

If you're reporting two or three digits, you will see the variation in those results.

So if you're not seeing variability, maybe you're not reporting enough digits in your results to see it.

And let me show you that in another way.

Some of you may be familiar with the concept of normal distribution.

So this is basically, I took those scenarios and I entered the values in a worksheet and plotted them.

And what it shows is on this X-axis, it shows the grams per liter of tartaric for my TAs.

And I show this to two digits.

So, the worst case goes from 6.6 to 8.4 grams per liter.

And my best case guy is right here skirting the 7.5.

This y-axis, the term density here isn't about the density of the wine, it's the density of the results.

So, how many results there are.

So I ran 30 days and here's basically number of results per day.

So if I express my results in two or three digits, I'm gonna see the difference between all these different curves.

But if I only express the results in a single digit, I'm not gonna tell the difference between my best case and my worst case scenario.

So that's the reason you wanna use some extra digits is that you'll be able to see the difference in your precision, okay.

So, that was our thought experiment.

What did we learn from our thought experiment?

I think there's four points.

The first is that even a perfectly performed analysis, you did it yourself, you followed the recipe, nothing went wrong, even in those cases, there's gonna be variability in your results and that the quality of your equipment and your supplies could have an impact on that.

But even with the best choices, there's gonna be variability.

Some of these errors are independent of any operator or any laboratory influence.

You're gonna expect there's gonna be actual variation.

And that's not counting meniscus errors, human error, random error, it's just built in this.

Also that if you express your results with fewer digits, you might be hiding that variation.

If I only expressed to one digit and ran the same analysis every day for a month, I might get exactly the same result.

But that's not because the results were exactly the same, it's because I'm not showing the degree of precision that would unhide that variability.

And the last point is, is that the true value is going to be in a range.

It's not a single number. That's the true result.

It's a range of results, it's a distribution of results that's gonna show you what the true value is.

So you can't say, I know this wine is exactly 7.5 grams per liter because there is no exact 7.5 grams per liter.

The result will be plus or minus whatever grams per liter is appropriate for the way that you can test.

So those are the four main points I think that we should have learned from the thought experiment.

- Can I ask you, - Oh, go ahead.

- [Molly] more questions from the chat, Pat?

- Okay.

- [Molly] So Drew says he does have an auto titrator, and he asked a question about centrifuge variability.

He says he centrifuges all of his TA samples.

So would that be something that could contribute to error?

- That if anything, would contribute to the volume of sample delivered, because if you pipette material that has solids, you're gonna have less actual liquid and more solids, if that makes sense.

So by centrifuging, you basically changed the volume delivered.

- [Molly] Okay.

- But again, that's probably beyond the scope of what we're gonna talk about today.

But that does contribute.

The main point here is that, there's error.

And now we can talk about what's this means and what we can do about it, okay.

What I showed you a few moments ago was a distribution of the results that might have looked like what we would get if we did our TAs 30 every day for 30 days, that we would see a different shape of a curve depending on how much error we had.

And that is what we would call a normal distribution of results.

And the way it works is that if I repeat analysis over and over again and I get aa distribution of where those numbers would be, we would expect the peak to be where our average is.

And that we would expect most of the answers to be clustered close to the mean.

And the further we get from the mean, the less of a cluster that we'll get.

So we get this very distinctive bell-shaped curve with 68% of our results will be within one standard deviation of this central average value.

And that 95 1/2% of the results are gonna be within two standard deviations of that mean.

And 99.7% of the results will be within three standard deviations of the mean.

And that gives us an idea of how often we would expect to see results between two and three standard deviations or above three standard deviations.

And most scientists like to kind of say, if it happens 5% of the time, that's something weird, you know.

5% is sort of a common number that we use as that's starting to be unusual, right.

So we would say that 5% of the time, we get something more than two standard deviations from mean.

That means there's something a little weird going on.

We expect 95% of our results to be clustered closer to the average.

So how can we apply this information to our 30 days of running TAs?

Well, first we need to know how to calculate an average.

And those of you who are up to speed on this, I apologize if I'm going too basic, but let's just kind of go through how to calculate an average.

An average is the sum of all your results divided by the number of results.

And in our thought experiment, in our best case, here's 10 example results, and I'm expressing them as many digits as I possibly can.

And our worst case results, we would add up our 10 results.

And in both cases, because we're testing the same wine, we would get the same average.

And then we would divide that by 10.

And in both cases, our average would be 7.5.

That's average.

How do we calculate our standard deviation?

Standard deviation is a little more complicated.

We have to take the sum of the difference between our result minus the average results squared divided by the n minus 1 and take the square root of that whole thing.

Now, if you're doing this by hand, you would have your 10 values for your best case and your 10 values for your worst case.

You would have subtract from that value the average that you would get for those 10 results, you would have to square that difference, and then you'd have to add them all up and divide them by, in this case nine because that's n minus one.

So that's pretty tedious.

But what we find out is that in our best case, our standard deviation is 0.028.

And in our worst case, our standard deviation is 5 79.

And that's only if you do it by hand.

And I don't know anybody that does it by hand.

If you go into your calculator or into Excel, you can just click the button or in Excel, you can just type in these formulas.

You say, give me the average for, and you select those cells or give me the standard deviation and select those cells, and that will get you mean in standard deviation like that.

So who needs to work all the way out?

All right.

So why do we care about our mean and our standard deviation?

We need those values to construct our normal distribution of our results.

So you see in both these cases, our worst case, in our best case, our average is the same, but our standard deviation is different.

Our less precise method has a higher standard deviation, and that is what we call a measure of our precision.

So precision again is if you look at that distribution, our normal distribution, these three curves are all normal distributions and they all have the same average.

The difference is, is the spread, the flatness of it.

And that is a measure of precision.

And this is specifically standard deviation, which we just calculated, is a measure of the dispersion across the mean.

So in our fictional best case, our standard deviation was about 0.03, and in our fictional worst case, it was about 0.41.

Other expressions that you might hear when you talk about precision are things like coefficient of variation or relative standard deviation.

These are a way of normalizing standard deviation by dividing it by the mean.

So that kind of weights your standard deviation relative to the mean.

It comes very important.

Because if I just say my standard deviation is 0.2, you don't know if that's important or not, unless I tell you my mean is 370 versus my mean is 2.

So you can have the same number of standard deviation, but how it looks on your relative to the values that you get will be relative to your mean.

So this is why they call it a relative standard deviation.

It's gonna be your standard deviation divided by the mean times 100.

So those are all measures of precision.

It tells you how flat your distribution is.

We're not gonna talk a lot about accuracy today because it is the subject of a whole nother can of worms.

How do we know in our thought experiment?

We just said we're gonna assume the true value is gonna be using a five mil sample.

We used 5 mls of 0.1 normal sodium hydroxide.

So we had a fictional true value.

And how do you calculate a true value in real life?

Well, in wine it's tough.

So you can buy reference standards where someone tells you what that value is, but how did they figure that out?

You can do a spike edition where you take your wine and you add a known amount of your compound and look at the difference between the spiked and the unspiked.

You can look at what a reference method gets you as a result, or you can look at the averages of high quality results.

Like if you send out 100 samples and you get 100 results from 100 good labs, maybe that's the true value.

And the way we'd express the accuracy is, we take that true value and we would make it relative to the result that we got.

So if I measured my TA and I knew the true value of that TA and I divided it by the true value, I would get a percent error.

But we're not gonna talk a lot about accuracy today.

We're gonna talk mostly about precision.

Okay.

We're gonna change gears a little bit here.

The whole point of our thought experiment was to understand the variability that would occur naturally in any analysis that we run.

But if we wanna talk about lab quality, there's a lot more things to consider.

So, if I look at the most basic six, five things, excuse me, that are important goals for laboratory quality insurance, there's a lot more than what we're covering here today.

So the first one would be looking at things that basically focus on the method and the accuracy and precision in a method.

And this would be method validation, which we're not talking about today.

There's a whole aspect of lab quality where you record your instrument performance, and that is a whole maintenance program that we're not gonna talk about today.

What we're focusing on here is maintaining a way to look at the results and understanding the accuracy and precision of the results.

So, we'll finish up our talk focusing on that.

And another thing to look at is the training needs, we're not talking about that, and how to do overall quality continuous improvement.

So if I break these five points up into sort of methods, instruments, monitoring, training, and continuous improvement, what we're really focusing on by looking at the accuracy and precision of the tests that we run all the time, it's what we would call the monitoring.

Okay.

So you guys remember our 30 day thought experiment?

What would happen if we tried to do that in real life?

If we did that in real life, we would have a pretty good idea of what the precision of our analysis is because we would see the natural variation that's coming, not just from our equipment but from, and we start factoring in other errors that might be natural to our lab that we didn't talk about today.

So maybe there's a temperature influence we can't see, or there's a water quality issue or a vibration thing I don't know about.

There could be, you know, the centrifugation of your sample.

There could be things that we just don't know.

And this is why it's dangerous to say, I know my method's good because I followed the recipe or the instructions because we don't know what some of the limitations are in your own laboratory.

So if we wanted to do this in real life, run the sample every day.

Our first challenge is that thing that we made it sound so easy is, we need a wine that's not gonna change over the course of a month so that we can assign the variability to our analysis.

And that's gonna be a challenge because if I just open a bottle, I am suddenly gonna have headspace there.

And we know that headspace could really affect some of our most common analytes.

So the ones that are particularly sensitive to maybe change from a partial bottle would be things related to volatility like alcohol.

Free and total SO2, it's not only a volatility thing, it's also an oxidative loss.

We can start getting into microbial metabolism issues like residual sugar or acetic and volatile acidity.

All of these things are a problem with trying to find a way to provide a wine sample that's gonna be stable over the course of time.

So the way a lot of laboratories deal with this is to try and find a way to minimize the concept of the headspace, either by single serve, where you're opening a fresh bottle each time, or you're looking at a larger container that is designed to reduce the impact of that headspace.

So the main choices that we see of people that want to run our 30 day thought experiment in real life by running what we will call a control, is to look at either buying boxed wines, which allow you to draw multiple samples over the course of multiple days with a minimum impact from headspace because the bags do control that.

Or alternatively, people look at buying single serve or single use, like a 187 sample.

And the biggest issue with all of these is, free and total SO2.

They're always a challenge because they're not only volatile, but they're oxidative.

So if you're running controls on SO2, that's the hardest one to deal with.

If you are gonna look at running this thought experiment in real life, running controls, if you do buy these samples, you want to look at production dates and make sure that they're contiguous.

Preferably if you're buying bag in the box, buy from the same pallet, from the same production time.

The same with the the small bottles, you wanna make sure they all came off a line at the same time, same date.

So that's the biggest challenge.

You just can't go buy a pack of four one day and go back the next day and buy another pack of four.

So if you wanna get started with controls and basically do our thought experiment in real life to collect your precision of your analysis, you're gonna want to buy enough of the same wine, preferably get something that you can run for several months, even up to a year.

How do you choose which wine?

If you are looking at, you only run white wines in your lab, then you wanna buy a white wine control.

If you only run red wines, you're gonna wanna buy a red wine control.

If you are doing both.

Best practice would be to do both.

If you don't have that luxury, we do see more variability in a red wine.

So if you want to sort of maximize your variability, you'd look at a red wine control, all right.

So now you buy the wine, what do you do?

Every time you run that analysis, you run a control.

So if you run a lot of analysis, if you run, you know, 30 TAs a day, then you're gonna wanna run a TA at the beginning of the day, and a control at control at the beginning of the day and a control at the end of the day.

and a control maybe every samples you run, put a control in there.

So every 10 samples is sort of a traditional number.

At the QC system we had at ETS.

You had to run a control at the beginning, the end, and every 10 samples, or you couldn't proceed.

You couldn't enter your data.

If you're not running that many samples, then every day that you actually run a sample, run a control.

That's sort of the minimum start.

Run at least a control sample on the day that you perform analysis.

Some people say, ah, but I only run one TA today.

Well, you have to run a control too, so you just doubled the amount of your analysis.

But it's a small price to pay.

All right.

So then you run this sample, you treat it just the same.

Pipette it just the same, titrate it just the same, and then record your result.

And you want to record that result with enough adequate digits so that you're gonna see your variation.

Okay. Then what do you do with this data?

I've run this every day.

I've got 30 analysis.

You're gonna need to track your results somehow.

Either you're gonna plot them on graph paper or you can use Excel.

Those of you that that have something like Excel can really save a lot of time.

And what is it that you're gonna look for when you plot the results?

We're trying to get to the concept of what is your in precision so that you know what is a reasonable variability in your results.

So now we're gonna get into what's known as control charting.

You've run your control, now you're gonna chart your results.

So I stole some data from a winery from many years ago.

And this is their TA control data.

So they have a data point and I again, let's just play with the month.

They have data point 1 through 31.

They have the date initials of the technician, what the result of their control test was and any comments that they made.

So this was something that they actually entered into the computer.

When they first started, they just had a piece of paper, but they rotated into directly entering it onto an continually open spreadsheet in the lab.

So you can see in this case they had TAs ranging from 6.54, oh, there's like some low values.

So it's kind of hard just looking at this to see your variability.

So you can get Excel to kind of take an average, a standard deviation and account.

So we can see Excel calculated the average was 6.45 and their standard deviation was 0.1113.

And you could plot that on a graph.

The X axis here is time, this is just the day, and the Y-axis is your actual TA value.

And then, there's what our average would be.

That's still a little bit hard to see what's going on.

You see that it ranges quite a bit.

But how do I know if I have a problem if something's not acting correctly?

If I have sample that one of those should only happen 5% of the time kind of sample.

So there's one more thing we can do and that's to normalize these results I've gotten in a standard deviation.

If I take my result, I subtract the average number and divide it by my standard deviation, I have what's known as a normalized value.

And what's nice about this is what this normalized value tells me is how many standard deviations I am from the mean.

So I can scroll down real quick here and say, okay, here's something that's 1 1/2 standard deviations from the mean, and here's one that's 4.3 standard deviations from the mean.

And then I can actually plot those, which means my average value is now a zero because it's the center and these are my ranges, my standard deviation ranges.

And why is this handy to use?

If you remember our normal distribution and our percentages within 1, 2 or 3 standard deviations, what this control chart is, is basically standard deviation, excuse me, a normal distribution turned sideways.

So I know what my average is and I know that I should have 68% of all my results within 1 standard deviation.

I should have 95% of all my results within 2 standard deviations.

And if it's more than that, I'm starting to get a little hairy.

And if it's more than 3 standard deviations, that should only happen 0.3% of the time.

So by looking at that, oops, by looking at that, I can see that this one result way out here, there's something obviously wrong, something went haywire and it was literally out of control.

So if you're wondering what the term out of control means, it means it's probably more than 2 or 3 standard deviations from your mean.

So here's that same lab.

This is what their actual alcohol analysis results look like.

So you can see how often you would expect to have results that were more than two standard deviations out of the mean.

Out of 100 samples, we would expect two or three times to get that because if you don't get that, there's something else going wrong.

You don't have a normal distribution of your variability.

And that's just as problematic as getting a problem, something that happens more rarely.

So you would expect statistically that if you run 100 samples, you're gonna get a few that are gonna be more than 2 or more than 3 standard deviations from the mean, but that most of them should cluster around your average.

And this will give you your true imprecision of your analysis.

Okay.

That particular winery that I'm talking about did go whole hog.

They built these nice spreadsheets that allowed them to continually track their control charts.

And this is something that you can build yourself or we can figure out if this is a value to you, different ways that you can go about doing it.

I guarantee that if you start calling up Molly saying you have questions about how to do this, she's gonna be ecstatic because she would like to see people feeling more comfortable running controls and understanding what it means.

There are entire like six unit classes on how to calculate and run your controls.

And we're not gonna have time to go through all the nitty gritty on what it means if you have more than three or four results in a row above the mean versus below the mean, what some of these results might mean from an analytical laboratory quality control standpoint.

But before you can get there, you have to start running them and dealing with some of the practical aspects of, well what wine should I use?

And does this work for SO2 for me?

Or I have a bigger variability than I thought.

Maybe I'm doing something wrong.

Until you start collecting this value, you can't answer that question of how much variability would I see if I ran it every day for a month?

And this is how you would find out.

So that is sort of the introduction of how you would understand the variability and then how you would begin to monitor it by running controls and tracking it by doing controlled charts.

And that would allow you to answer that thought experiment question with a little more, what's the word I want?

Authority on what the actual variation that you get would be.

So I know I've droned on for quite a while.

We have some time now for some questions.

And I can stop sharing.

Hopefully everybody's awake.

- Hey, Pat. Thanks for a great presentation.

So I'm looking here in the the chat and we have some additional questions.

- Great.

- Isn't another component, and you kind of alluded this, but isn't another component that contributes to the testing variability, the environment?

- Oh, certainly. Oh, yeah.

So that's the reason why even when you look at a published method and a really good published method will tell you what some of the variability is in your method, but when you apply that same method to your own environment, it could be different.

So what a method validation study would be is telling you in the best conditions what your in precision would be and then you have to apply that to your lab and measure what it actually is in your environment.

That's kind of like, I know my oven is colder than what they use on the television.

That's part of my environmental issue.

So that means I can't just follow the recipe and use 325.

I have to set mine for 350.

That's an aside. Sorry.

- It's applicable.

So someone had asked if this presentation will be available afterwards?

Yes, it's being recorded.

and it'll be posted onto the extension website probably early next week.

So you can watch it again.

Are there any other questions?

I think I got them all, Beth.

- Yeah, I think so.

We just kind of had some commentary in the chat.

Molly, by the way, is there a survey that for this, that you needed feedback survey?

I can't remember.

- I don't remember if I came up with a survey or not.

- That's okay.

I'm just jogging your memory so that, yeah, yeah, yeah.

It's fine if there isn't.

Yeah, you know Pat though, I mean if we do have one moment, I was curious, you know, you're talking about the free and total SO2 and that kind of bouncing around or could bounce around.

Would there be any specific, I don't know if this is getting into anything proprietary, box wines that you would recommend as being really great or even certain varieties to move towards or anything like that?

And any tips for that control?

- Yeah, if we want to talk about, I think we need a whole nother talk because there's so many factors with sulfur dioxide and even with my experience in producing box wines, the variability in the packaging, the variability in the wines, it's incredibly complicated.

And that's part of the challenge that we have.

I think really to do a good job with really understanding SO2 we'd have to manufacture model solutions.

We'd have to, instead of using a control wine, we'd have to have just standards.

And even making those can be very challenging.

Sulfur is a tricky one.

But you know that, right?

- I do but I don't know it as well as you do.

And I wasn't sure, you know.

I'm constantly trying to think of ways to bring on this to the practical, you know, level of take home tips, you know, which I got a lot from here as far as the quality of tools and how that can help you or hurt you kind of.

Yeah, yeah.

- I would say with any of these things you just buy your box wine and start running SO2.

Cause until you start doing that, you're not gonna know anything more.

And some of these boxed wines will hold their total SO2 pretty well depending on how old they are, whether their free SO2 is fairly stable.

But until you start doing that, you're not gonna know.

So if you run it for a month and you see a slow decrease over time, that's telling you something different than if your noise is so high that it's gonna outweigh any natural changes that you might see in the wine.

If we look at truly what the industry performance is for some of these tests, it's higher than you'd expect from a wine changing over the course of the month.

So, I would say just get started.

And as you get the data then I'm sure you know, you and Molly would be happy to have a whole seminar talking about the results that people are getting.

- Sure. Yeah.

It's also always, I think, fascinating to look at some of these varieties as Anne Sandbrook alluded to when you're looking at the grape primary fruit chemistry and you know, some of the interferences from that juice matrix and how that can cause additional challenges with pH and TA, yeah, with getting really honed in on that accurate and precise number.

- So I couldn't find the survey.

So what I'll do is I'll email it to the participants because it's very important that we have your feedback.

So I'll email that to you probably by the end of today.

And if you wouldn't mind taking five minutes to fill that out, we'd really appreciate it.

- Yeah, I'm sorry we drowned for so long, but it's a lot of material to cover it.

Hope you guys stayed awake.

- We're awake.

- Okay. Yeah.

- Yeah. But you guys are glab nerds.

- We did have one question and you sort of just said it Pat, but I'll just, you know, just for the sake repeat it, for a control wine, would it be better to make a model wine was kind of the question that just came in?

- So there's gonna be a difference between making a model wine and a control wine 'cause the matrix in wines is so complicated that it's, a control versus a model are gonna be different.

So you would wanna use like a model wine when you're validating your method.

So, that's part one of the five parts of a lab system.

Once you accept that method is being something you already know the parameters for, the control wine will help you understand the impact of matrix.

So that's why if you are running red and white wines, you should be running a red and white control.

So yes, there's certainly a value to having a model wine, but not necessarily as for running a control.

- Makes sense.

Are there any more questions?

I think that's it.

- Well sure. It's nice to see such a good turnout.

That's quite a few people actually showed up for this.

- Well, thank you Pat for you.

So Beth and I thank you as well.

Thanks a lot. - Yes, very much.

That was great.

- All right. Well, enjoy yourself, you guys.

We have a cold snap coming over here, which means it'll be over for you guys in about a week, right?

- Thank you.

- All right. Yeah.

Nice to see you guys. Nice to see you Beth.

- Nice to see you. - Bye Molly.

- Thank you so much, Pat.

All right. Take care.

Bye.