Harvesting Innovation: Efficiency Through Orchard Automation

- Hi, everyone.

I see the meeting room is populating right now.

My name is Jim Ladlee.

I'll be one of your moderators, along with Dana Ollendyke today.

Before we begin, or while we're waiting for people to jump on board for the webinar today, one of the things that we'll be using throughout the day is a audience response system called Slido.

And I'm gonna move to the next slide here, and just, or share my slides, and so you can see what I'm talking about.

So while we're waiting for everybody to join the meeting, here's a question for you.

It's pretty simple.

But we'll be using this throughout the day.

I think we have five or six questions we'll use this for later on.

But a simple one and an easy way to get started with the conversation is simply, we ask just which state or province you might be joining us from today.

And all you have to do is scan that QR code that's on that slide, or you can actually join using your web browser by going to slido.com and just entering in that code, and it would allow you to enter in your choices, so.

This is more about getting a feel for how this system works, but always nice to see where people are joining from across the United States and around the world sometimes.

So that's always great.

Yeah, Bueno Aires, welcome.

Welcome from Connecticut and Kentucky and New York and Pennsylvania.

Looks like we have one more person jumping in.

Okay, I'm gonna go ahead and get started so we have as much time as possible for Dr. He today because we literally have one of the premier presenters today, so I'm really excited about the webinar today.

As I said, your moderators for today will be myself and Dana Ollendyke.

I'm the state program leader for emerging and advanced tech with Penn State Extension, and Dana is a Penn State Extension program manager who's responsible for all the good things that happen on these webinars, by the way.

As we're going through the webinars, there's actually a Q&A and a Chat function.

For the Chat button, if you're having any technical issues, something's not working right, you can't see something, put it in the Chat button, and we will try our best to resolve that as soon as we possibly can.

As with most of these webinars, it's not always possible, but we will definitely try.

And then, if you have a question for Dr. He as we go through the presentation or any of us, feel free to put that in the Q&A, and we will either address those live online.

As long as time presents, we'll ask as many questions as we possibly can, or if we can't, we'll get you a response in writing after the webinar is over.

Again, going back to the audience response, the Slido presentation here.

When you think about orchard automation, what one word comes to mind?

So again, this helps frame the conversation for Dr. He as we go through.

So if you wouldn't mind going back to the Slido audience response system, and when you're thinking about it, and just answer that question.

Just one word, that's all we need.

When thinking about orchard automation, one word that comes to mind is robotics, efficiency.

Oh, a couple of efficiencies in there.

Labor management, quick, precision, information systems, excellent.

Give about 15 more seconds for anybody who wants to get a response in there quick.

Again, you can do this online or by scanning the QR code with your phone.

So I'm actually online myself with this, so it's pretty easy to do either way.

So you can just scan it with your phone, or you can join on a web browser and be able to put your responses in there.

If you wanna stay in there for just one more second in the Slido, here's another question for you, and this one's pretty easy.

You just have to select one choice.

It's basically, I would rate my current knowledge about efficiency through orchard automation as, and then you just pick one of the categories and hit Send, and that's it.

So these will not automatically jump onto the screen, but we will be sharing those in just a second so everybody can see the results.

Give you about another 10 seconds to just answer that question.

All right, we're gonna move on.

Hopefully, everybody who wanted to had an opportunity to jump in.

All right, that's actually really valuable information, so thank you for everybody who participated in that little quick poll.

We have a couple people who are experts, and a couple people who have very little knowledge, and a lotta people kind of in the middle somewhere.

So that's very, very useful.

Actually, as you just saw on that poll, you can continue to add that information in even after we move from this slide, so the poll will remain open.

So thank you very much for doing that.

All right, a little bit about the Harvesting Innovation Webinar Series.

Our goal really here is to promote collaboration, conversation, and learning around technology.

It's really to bring people together as we have these conversations.

We're trying to share emerging technologies that can be integrated into real-world applications.

And then, we're trying to share technology to support efficiency, sustainability, and competitiveness, and profitability.

So again, if we can even come close to those three goals during our presentations, we're very happy.

And as a result of that, one of the things that we have is we have with us today, we have Dr. Long He, who is an associate professor in the Department of Agricultural and Biological Engineering and is located at the Fruit Research Center, Fruit Research and Extension Center at Biglerville.

He received his PhD in mechatronics and engineering, mechatronics engineering from Yanshan University in China.

Before joining Penn State, Dr. He worked as a postdoc research associate, research engineering at Washington State University and the University of California at Davis.

Dr. He's research interests are agricultural machinery and automation, including mechanical harvesting technologies for tree fruit crops, robotic solutions for agricultural applications, and sensor applications in agriculture.

Now, that's Dr. He's formal introduction, but I can tell you on a personal level, having had the opportunity to travel around the United States looking at different ag innovations and technologies, he's one of the best faculty members that I have found anywhere in the United States, so I'm very excited about his presentation.

The work that he does at FREC, at the Fruit Research and Extension Center, is really cutting-edge and phenomenal work.

So I'm super excited to be able to bring Dr. He to the Harvesting Innovation Webinar Series today because I think he has insights on what's coming in the future and where things are headed and some of the great work that he's doing.

So, with that, I am going to stop sharing my presentation, and I am going to turn this over to Dr. He, so.

- Okay, thank you very much, Jim, for the introduction, and thank you for you and Dana for inviting me to this webinar.

And I would like to thank everyone for tuning in for this webinar.

So today, I'm going to talk about some efficiency through orchard automation topics.

That actually is, in the past seven years, I was at Penn State and working on orchard automation, so there's different projects I'm going to go through today.

So, like Jim said, I'm located at the Fruit Research and Extension Center in Biglerville, PA, so it's about 2 1/2 hour down south to campus.

And so, first, I would like to very briefly just talk about my research program.

So here is the current members in the lab.

Right now, we have some students and postdocs.

And our lab mission is to, the whole goal is to develop an integrated precision agriculture research and extension program here.

So we do research on sensing-based crop management and decision-making and robotic solutions for specialty crop production.

And also, our extension is a part, is one of the big part of the program here as well, so we do extension for diverse specialty crop growers in terms of adopting technologies.

And also, we would like to take our research position to be a agriculture innovation node to connect to various stakeholders, including farmers, tech companies, and then government agencies, and others and professionals.

So, so today's presentation, the work that was done by, submitted by the students, at least here, but also for some previous students already graduated from the program and a lot of my collaborators in the past few years.

So, for our program, I'm trying to see what we are targeting for, so what kind of challenges that we're going to take on.

So here's our list of four major kind of area that we are thinking that our research and extension effort can put into.

The one is the labor.

Labor, especially kind of labor shortage, is a kind of urgent challenge for the agriculture industry.

I mean, for most of the agriculture industry, of course, the tree fruit industry is one of them.

And labor needs throughout the season also is not in balance.

So it's sometimes, in the peak season, you need much more laborers compared to other seasons, so they're unbalanced in a season, and how to address that problem.

And also, the increasing cost for labor is another challenge for growers to maintain their profitability.

Efficiency is another one.

So, for row crops, for example, there's a lot of automation and mechanization already happening, but for specialty crops, there's still, most of the practices rely on manual operations, for example, pruning and harvesting.

So need a lot of labor to be involved in those kind of practices.

And then those can lead into low efficiency and also some of the practices as less, kind of inconsistency due to the human subject, kind of different.

We have kind of practice differently from one to another.

And then, the precision is another thing.

We try to also target this precision.

Precision here is more like how we, mostly, it's like how we precisely apply agriculture materials to the plants, to the crops.

So right now, a lot of practices are just using fixed rate, or some are causing a lot of off-target kind of spray or application.

So it's in a low precision, and this can lead into the high cost of the production and some of the waste of the materials.

And eventually, want to talk about sustainability.

Sustainability is more like the integration of the, I would just mention about the three above.

Like if we address those labor, efficiency, and precision issues, then we are actually leading to a very sustainable agriculture systems.

And then, that can lead into economically as more like we're kind of reducing the farm input cost and then also gain more from the output from the farming.

And then, environmentally, it's more like if we use less water and less chemical, that actually bring benefit to the environment as well, and societal impact that we as a human can live a better life with clean environment and also affordable produces.

So those are the challenge we're gonna target.

And then, what are the emerging technologies that we are going to kind of look into, or some of those are, like, available at the moment.

Here are just some examples.

These are not all of the emerging.

There are so many different areas has emerging technologies, but in relates to our applications, I list a few here.

So first one is robotics.

Robotics is more like everybody think about the automation, so it's kind of somebody, some system that can replace human to work on a farm.

So the robotics mainly including kind of perception systems, so you have manipulators, end-effectors, and it can be also aerial or ground-based robots to conduct different operations and involve path planning, navigation, those kind of things.

And then, remote sensing is another one, or generally sensing part.

We have sensors and satellite imaging or UAV-based imaging to work on the crop monitoring to have better sense of what happening in the field.

And then we have then generated the application maps for management.

Our next one is Internet of Things.

Internet of Things is another technology that can be used for connecting your crops, connecting the data, and also, and use this together.

And then, it's more like it's connectivity, communication-based technology.

And then, of course, there are sensors involved to monitor the thing that we're interested on and data.

There are also cloud computing, decision support that help us to better manage in the field.

Another one, I want to mention about the variable rate technology.

So, normally, we have orchards or a field that has a certain size, and then there's a different, there's big variations even within one field, or the trees or the crops are different size and everything like that, so we like to look into the difference in the individuals, and then to see if we can apply something there more precisely.

And the variable rate technology is one of them, and so it's more like site or plant-specific.

And then, they also involve control system and take actions.

And AI models is more like in recent years, and then AI has become very hot topics for all kinds of dominant area, and agriculture definitely is one of the big application area.

So we have talked about machine learning, deep learning, and then generative AI and large language models.

Those are the potential applications for agriculture.

So all of those emerging technologies actually is putting a lot of, put a lot of things into the agriculture area that makes this process getting easier, so we're definitely working on those and see how that it can help.

So today, I'm going to just go through some of our projects that have been worked in the past years.

And then the first one is like sensor-based irrigation, and we have some spraying technologies, and then robotics, especially for the crop load management for tree apples, for apple trees.

And then, I'll very briefly talk about extension program that I have been involved in in the past.

So, for precision irrigation, you see here, we have been testing actually different technologies to see how to help irrigation, and soil moisture sensors is one of them that we found it's kind of easy to implement and also pretty kind of affordable for growers to implement and to apply.

So, we actually did some experiment to install sensors into the orchards and then to monitor the soil moisture level and then to apply irrigation based on those data.

Here, it just shows some pictures showing how we installed the sensor into the ground.

And then, you see, though, we have different depths of sensors, and then we can monitor the soil moisture in the hole depths of the root zone.

And then, the data will collected by a data logger, and it can be transmitted to the end users, and then you can access the data.

So here are just some primary results we got for like, how the sensor-based irrigation can help us to do the irrigation and actually saved a good amount of water for our irrigational season and other things.

Like, and we also maintained a pretty similar kind of crop yield and crop size compared to just for the conventional irrigation.

Conventional is more like based on experience.

Like we normally irrigate once a week, for example, those kind of things.

And ET is called evapotranspiration-based irrigation.

And then, so we did a comparison, like soil moisture in the ET-based.

So we found that ET, actually, the sensor-based actually saved more water.

So it's kind of like based on the need of the water that we applied the water to the ground.

And then, for those, for the first one I mentioned about there is more like using sensors to monitor the soil moisture, and then we can apply water accordingly.

And then, later on, we thought, think about like how we could actually make this process automated, so we developed this automatic irrigation system.

And then the sensor's here, put into the ground, and then connect to a control box.

There's actually a solenoid valve that's also connected to the box.

So, within this interface, on the right side, you can see that the data from the soil moisture can be recorded in real-time, and then we can actually set up a threshold to see when we should start irrigation.

So the numbers of the soil moisture, the value of the soil moisture can give a signal for the valve to be turned on and off.

And then, we can also see this, just this on the bottom, it's more like indicating if the valve status is on and off and also can show how this soil moisture in the ground actually changes through irrigation.

And then, after that, we actually did some field trials with different like crops and different situations.

Like, we did actually test this with commercial farm tree fruit growers, and then we installed sensors in their orchards.

We also put this test in high tunnel and open field and also greenhouse vegetable field as well to see how this potentially can work for irrigation.

So that is more like the automatic irrigation.

And then, we also think further step, like can we integrate AI with IoT together to help monitoring crops that actually potentially for water and nutrient management.

And this is a project, I worked with a vegetable specialist.

And then, you can see this is just a diagram of how we're actually putting these two technologies together.

We have sensors in, put sensors in the greenhouse, and then connect with a kind of like camera and Jetson, kind of an AI-based microcontroller, and the data can be transmitted to the cloud, and then we actually can obtain the image data from there.

Then, we can process data to get the crop's status from the greenhouse.

And here's is the setup that we got.

You can see that the camera was put up above of the crops, and then you can just continuously monitoring the growth of the crop.

And then, also, sensors are put into the solutions, and then we can actually connect all of them into the controllers.

And those information, the data and images are collected, and then, through the cloud, we can actually remote access to those information.

And afterwards, we developed AI-based models to process the data.

And then here, you can see from the beginning to the middle and the later stage, we can see, we can actually see how, segment, actually, each individual plant, and then we can actually calculate the size of the plants.

And this is actually a good indicator of how the plant's growing throughout the days before harvesting.

And this is a good indicator for later on if we are connecting with the nutrient management, and we know and we understand what is the time that we actually need to apply nutrients to the plants.

And then, and also if we put this system in an open field or in a soil-based system, we actually can control or also can use this as information for irrigation as well.

So that is more on the irrigation.

And the second one I want to talk to is a advanced spraying technology.

And I want to mention that spraying is a very, very important practice for tree fruit orchards, especially in the northeast region.

We have a lot of disease and insects pressure throughout the season, during the humid weather conditions, humid weather conditions here.

So, conventionally, the growers are using conventional sprayers just to spray as the presetting.

Like they each, for example, normally, we set at 100 gallons per acre, so that's something, once we set up, then we just go through the orchards and then just spray.

There's no kind of adjustment throughout the field, and then just spray at the fixed rate.

So the concept of variable rate sprayer is more like we look into the objects that we're gonna spray, no matter it's a tree or flowers or fruits, and then we actually are using sensors to calculate the needs of the spray rate.

And then, throughout the orchards, we can using this method to apply more precisely.

So here is the principle.

You can see that, normally, using sensors to identify the objects and then process with this computer.

And then, with all the models, we can calculate how much kind of chemicals we need to apply to the objects, and this could be controlled.

For this example, we're using individual nozzles, so we can control the nozzles individually to apply to the canopy.

I will talk more about this later for our application.

So this one is a example of a variable rate sprayer.

So actually, here, you can see, on this sprayer, there's a LiDAR sensor.

There's a GPS on top of that, and the nozzles could be individually controlled by solenoid valves.

And this one has a app to operate as well.

You can record how much chemical is applied to the field.

And then, these are two field tests we conducted, and we actually test this in two different tree structures.

One is just conventional kinda, you see the big canopy of the tree, and the other one is high-density.

It's a smaller tree, but more in dense, more dense growing conditions.

So, we are also targeting on different diseases, and then we test with comparison with conventional sprayers.

And just briefly, we got around 50% of chemical saving throughout the season.

In the early stage, you can see, when there's not too many leaves, the saving actually can be more.

And later on, when the canopy grow up, the tree's getting more dense, and then the saving reduced a little bit.

But still, this type of technology saved around 50% of the chemical throughout the season.

And then, so even with that, with this, when we test this, especially at a early time, we still see a lot of chemicals actually passing through the tree, and then it's go through the tree to the other side, and those chemicals, that actually could not be used.

It's called chemical drift.

So we are thinking about what way to actually reduce the chemical drift because chemical drift is not only wasting chemical, but also, it's not good for the environment.

It just kind of drop to the ground or would fly into the air, so it would be a very bad kind of environmental impact there.

So, we think about that, and we proposed a method actually for spray drift reduction.

So this is, you can see, for the left side, you can see there's a sprayer that we add another layer at the end of the sprayer.

It's called airflow control.

So airflow control is using this.

You can see in the middle there, it's called iris dampers.

The opening can be adjusted.

And, of course, there's LiDAR there that I use.

We are using LiDAR to sense the canopy.

Like, as we discovered earlier, that if the canopy is narrow, is less dense, then the drift is more.

So we actually think there is a good relationship between the canopy density and the drift, so we're using LiDAR to measure the canopy density.

And then, we use that density to control the iris damper opening, so to control the airflow.

So here, just on the top here, you can see three different examples of openings.

We have like the largest opening is kind of full open until very small open to adapt into different canopy density.

And for this test, we're actually using water-sensitive papers put on different location on the tree, but also, we put some on the ground before the tree and also after the tree.

There's also a few there trying to measure the drift of the spray.

And after a round of, after many kind of tests, and then we can get this model.

It's like the model is between the canopy points and the damper opening, the iris damper opening.

So it's very straightforward.

Like, for example, you measure the density of the canopy, you know, the number of points from this tree, and then we can have corresponding opening of the iris damper to control the airflow.

So this is the model we got.

And then, afterwards, we can use this model to control the opening as needed based on the new orchards that we're going to test.

And here is a one test example.

So after we got a model, we evaluate the model, and then we go to an orchard block.

And then this is how we got the results.

So you can see, the black one's the ground, and then this yellow one is the next row.

This means those two are drift.

On between those are, or not this tree.

I mean, this black one and orange one are the drift, and then the between those are on the tree.

So you can see that, actually, the drift through this, after this kind of airflow control, the drift significantly reduced.

I didn't have the comparation here, but I can see the drift is very, very minimal, and most of the chemical actually deposited onto the tree canopy.

That is what we expect, and we would like to see.

And the sprayer, that variable rate sprayer, we still need a tractor to drive through.

And then the next one I will talk about is called unmanned ground sprayers, so that means we don't need anybody to drive the sprayer.

The sprayer can drive itself in the orchards.

And then, this, how this works, this is just a short video.

You can see that this is a sprayer, is a unmanned sprayer.

It's a ground sprayer that actually can drive through the orchards with presetting map.

So if we can map, we map the field first, and then we get the path.

That can be imported into the sprayer, the sprayer can drive through the orchards by its own to spray as what we kind of set up in the beginning.

So this definitely brings some kind of benefit.

So we don't need to have a operator close to the sprayer, close to the chemical, so it's reduced the hazard to the human.

And also, using this, we can actually increasing the spray efficiency by kind of, like you can see this.

This sprayer has two nozzles on each, each side has one, and then we're also trying to spray only to the canopy range.

So this is how it works.

So this sprayer actually has two kind of adjustment on the angles.

One is horizontal.

You can see this.

The angle can be adjusted at a different angle to the tree rows and also, vertical angles, so that is up and down, so that means the cover range could also be adjusted.

So this is how we conducted the experiment.

This one, just put some water-sensitive paper on the tree, and then we measured, like this coverage on those papers afterwards, after the spray, and this is like the results we got.

We actually test a different driving speed and also the coverage.

So you can see that, also, this is the angle that the horizontal angle that are actually facing to the tree.

And then, you can see, 90 degrees, that's actually directly towards the tree is getting higher coverage percentage here.

So that is because tree rows are actually pretty wide, so this spray on the middle, and then, like perpendicular tree rows, it actually can get a higher coverage compared to a little bitty angle here.

So this is a robotic kind of unmanned ground robotic sprayer.

We're still working on this.

And then we're trying to actually integrate these two technologies together.

One is the canopy measurements and also the autonomous driving together.

So eventually, the unmanned sprayer, unmanned ground sprayer can adjust spray rate as well as the angles based on the canopy appearance and density.

And next one I want to talk about, the project that we actually put a lot of effort on this.

And then, it's actually, it's a serial project.

It's not just one, but all addressing similar kind of problems.

It's called crop load management for apples.

So eventually, we want to have good quality fruits at harvest, so, for apples, it's like about six months, for example, five or six months of growth stages.

And from dormant, kind of dormant pruning stages to blossom thinning, to green fruit thinning, all of those are the practice that help us to get the final good fruit quality and yield.

So we have been working on all of those kind of steps and trying to develop precision and also robotic solutions for this process.

Right now, most all of those are actually, like, for example, pruning are handled by manually.

And the blossom thinning and green fruit thinning are, right now, it's mostly for using airblast sprayer to spray chemicals, or some are using manually to remove some of the crops on the tree during these stages.

We are trying to work on using different robotic or automated solutions for that.

So first, I want to just quickly look into, like, for those steps, what we have been working on.

So, for dormant season, for robotic pruning part, like you can see that here, we mainly worked on like mechanism, like end-effectors.

Here are the cutters.

How we could actually effectively cutting off some of the branches from the trees to do the pruning.

And most of our work has been done for design different end-effectors.

Like, there's cutters here that actually can reach to different orientation of the branch.

Because, in the orchards, the branches are growing very randomly, like different orientations, different size, so those requires very kind of different needs for the end-effector.

So we actually developed two type of end-effectors, trying to cut the different-sized branches and the branches at different orientations.

And also here, on the right side, we also put these end cutters into a kind of, this is an industrial robotic arm, to see how we can use this type of robotic system to prune the branches off.

So right now, we are still working on this for, especially, we will integrate more in machine vision, kind of how we can see the branches and then do the automatic pruning.

So that is probably the next step.

And once the dormant season has passed and then the flower buds start to break, and then we are looking to the bud stages and see like if we can estimate the crop load at the early stage, like, to see how many flower buds actually are on the branch, on the tree.

So here's is how we did.

So we actually collect a lot of images during different stages.

You can see these are from silver tip to tight cluster, and those are the flower buds actually.

Eventually, they're gonna turn into the fruits.

So it's very kind of in the early stage that we can count number of buds to estimate how many fruits we're going to get for later.

So, it will provide very useful information for farmers to evaluate like how many crops they're going to have.

Do they have sufficient pruning, or do they need to remove some of the buds, or how many kind of, how much thinning work they're going to do in the later season and later.

So this is actually some good information for that practices.

So we did actually acquire images and do the AI kind of, we developed AI model training to detect the flower buds on the tree and count number of buds.

And also, our students are also working on calculating the diameter of each branch.

And then because a certain size of branch can bear a certain number of fruits, and then that actually can relate together, kind of integrate together, like how many flower buds we should put into a kind of, into a individual branch.

That actually help us to, for example, if we need to remove some of the buds off from branch, that it can be done at this stage.

And then, once we are passing the bud stages and then we have kind of blossom stage, so the flowers will start to grow.

And then you can see that, for each cluster, like each bud, it's going to have five, six flowers.

And then the flowers are not opening at the same time.

So, normally, there's the center, the center or called the king flower that opens first, and then their latter flowers open afterwards.

So this can be open in a few days or in one day, so those depend on the weather, so it's a dynamic process, the flower opening.

And flower blossom thinning is another practice that can be used for crop load management.

If we have a lot of flowers, that means we're going to have a lot of fruit set afterwards.

So, at this stage, we can use blossom thinning to remove some flowers.

So, conventionally, we're actually using conventional sprayers to remove or to kind of do the chemical thinning.

We're trying to use more precision ways.

Here's we developed a targeted base, kinda targets, a chemical thinning system, using camera to locate the flowers, and then using nozzles, move to the kind of, to the flower cluster to do the blossom thinning to remove some of the, kind of, prevent some flowers to be getting into the fruits.

And this, the bottom one, is a newer kind of later version that we used, put on an autonomous sprayer that can do, autonomous platform, and integrate together for a kind of autonomous driving sprayer to do this work.

And later on, if we still have too many fruits on the tree, and then, during the green fruit stages, normally, the fruit's about 10 millimeter to 30 or so millimeters, and then, normally, before 20, we can use chemical, right?

So it's 10 to 20, for example, millimeters on the tree.

And you can see that each cluster have four or five fruits, but normally, we only need one or two, so we need to remove some of the fruits.

And then, we actually developed a series of the algorithms kind of to detect fruits, identify the fruit's orientation, and to do the kind of stem and fruit pairing and also clustering.

And all of those actually help us to make decision which fruits we should remove.

We don't want to remove all of the fruits from one cluster.

We want to remove, like, we need to remain, for example, remain one or two fruits in one cluster.

That's why clustering is also important to make sure we are removing a few from a cluster, not all of them.

This is the robotic system that we developed to do this, the fruit removal.

You can see that the end of here is an end-effector we actually can engage into the fruits to remove the fruit with this robotic arm.

And I would like to also, that's a robotic system for crop load management for apples.

And in our lab, we also worked on other project related to the robotics.

This one is for robotic apple harvesting I wanted to mention.

So, on the left side, you can see, this is actually something we did in the lab here.

We developed this Cartesian robotic system to kind of pick the fruits, and later on, we're also using machine vision system to identify the fruits and fruit stem and the branches associated with that and to better positioning the robotic end-effector to the fruits and then to pick more efficiently.

And those two here are the end-effector we developed for this purpose.

So we're still working on this.

And I want to also mention that we at Penn State are also involve a project that leaded by Michigan State and also USDA ARS in Michigan.

We had USDA's SCRI grants working on integrated robotic harvesting system with in-field sorting as well.

So this is the prototype the Michigan State has developed.

It's using a vacuum system.

So this will be extensively tested in both in Michigan, Washington, and also Pennsylvania.

So, hopefully, we'll have more results come up in the next couple years.

Another project we have been working on is robotic mushroom harvesting.

You can see the mushroom growing like this, like growing pretty close together.

And then we have been working on both machine vision system and picking end-effector mechanism to kinda harvest mushrooms.

And then those are the detection results you can see on the right side.

You can see the mushrooms.

And then, the bottom picture shows a decision support system developed by our students, working on how we can effectively picking mushrooms at different orientations that actually can avoid collision with others.

And on the right side is a prototype that we developed for picking mushrooms with a vacuum system.

And we also have some research on using UAV-based for crop health monitoring that, for orchards, we fly UAV and are using a multispectral camera, for example, to generate the map for the healthy map of the orchards, the trees, and then we identify if there are any kind of disease pressure happening in the field.

So those are still very preliminary study.

We would like to put more effort into that in the future.

And also, trying to quickly look through some extension activities.

We occasionally host some workshops.

So, for example, we have, these two are showing some of our sprayer system that we're actually trying to engage growers to look into how the system works.

And especially if they have interest, they can have a little bit of hands-on, those kind of workshops.

We also do some commercial field trials.

These are more like the irrigation systems set up in different commercial orchards, on a commercial field to test the irrigation system.

And we have agriculture field day at Fruit Research and Extension Center.

We have like, every other year, we have field day, a grower field day.

And also, on the other year, we'll also do a precision app technology field day to actually invite in some of the tech companies to demonstrate their product and also showcase our research as well.

So those are some of our research project.

And then look into the future.

I would like to, like, we would like to understand more about diverse industry needs because the farmers have, like, large-scale, small-scale farmers, they may have different needs.

And then, how those emerging technologies can help for different needs from the diverse growers.

And also, investigate some biological system challenges.

For example, tree orchards has a very complex environment, and how we can actually investigate more into those and how we understand it better for those biological system challenges.

And develop and test emerging technologies, solutions with integrated system, and also build network and collaborations within Penn State and outside.

And eventually, we would like to enhance the technology adoption by growers.

So that's everything.

Thank you so much for listening.

And then, if you have any questions, comments, I would be happy to answer, thank you.

- Thank you, Dr. He.

That was outstanding and even better than I expected.

So you have so much going on, and, you know, one of the great parts of my job is being able to have the opportunity to work with and learn from great researchers like Dr. He.

And his work really is phenomenal and helping lead us to the future, and where you're in an area where we can find technology that works for growers.

Dr. He, I'm gonna direct you to the Q&A because I think there's several questions in there that you are uniquely qualified to answer.

So we have a couple of questions in there, and if anybody else would like to add any questions to the Q&A, feel free to go down to the Q&A button at the bottom of your screen and ask questions, and we'll try and answer as many as we can before our time expires here.

So, Dr. He, did you wanna just jump in with those questions? - Okay, sure.

So the first question here is, "Are those irrigation sensors like old clay ones that we used with soil aggregates and handheld Bouyoucos moisture meters?" So those sensor, the sensors I list here is a little bit different from that.

So these sensors are actually, what we used is actually, we put it into the ground, and then it will stay there for a few seasons.

So we're actually not, don't take them out.

Especially for orchards, we just put them, stay there.

For vegetable field, we actually take it out every season because the vegetables, these need to renew every year for the plant year.

For the tree orchards, we just stay there for a few years to continuously lock in the data.

And the second question here is, "Are those stationary field sensors inputting weather data, wind direction, speed information, adding to the LiDAR data to control drift?" Yeah, thank you so much for the question.

I think this is something we're actually thinking about, thought about in the past, but haven't got a chance to test it yet.

And there's a little bit difficulty, sometimes, because, for the sprayer, for example, it's the tractor drives in a certain speed.

It's kind of very fast, dynamic process.

And then the wind direction and speed could be changed and very fast as well, so the sensors recording those need to be very, very sensitive.

So, we're still looking into that.

I think it's a very good idea to add the wind direction, speed information into the sprayer.

It actually can help us to kind of, to kind of to control the drift, for if the wind speed is too high, and then we probably won't go, but if the wind speed, we know wind speed and directions, we may be able to add some mechanism to control the spray directions and those kind of things.

But that's something, I think, we really need to look into in the future.

"Are the UAV camera's high-spectral range for pathology inspections, high-spectral range, or LiDAR-based technology?" Yeah, so the UAV cameras that we used is a multispectral, so it have five, like, spectral band.

So it's not a LiDAR-based technology.

LiDAR is more like measuring the reflection points, so we can actually measure 3D information.

But that like, more like multispectral, spectral imaging, even some of the UAV cameras using hyperspectral data can get more wider band of the images, so that can help us to find disease problems before our human eyes can see.

So there's two different things, I think.

Yes, we used a multispectral camera.

Thank you so much for the questions.

There's more questions.

"For robotic fruit picking, how far apart do you think the tree rows will need to be?" So currently, a lot of recommendation either from horticulturists or from engineers, I think it's about 12 feet apart from the tree rows.

That is pretty standard for high-density tree orchards, and then that actually is sufficient for robotic harvesting.

For example, that means we have robots driving in between the tree rows, and there's arms actually on each side to operate.

And then, mostly, the most important things for robotic harvest is the size of the tree canopy itself, like the depths of the canopy.

So, ideally, the canopy should be narrow.

If it's a conventional big tree canopy, it will be a little bit of challenge for robotic harvesting.

And next one is, "For chemical thinning, lasers may be an option, similar to what going on in ground-based specialty crops.

For chemical thinning, a laser may be an option." So, yes, for chemical thinning, so here's the, of course, so when we look at thinning, if we only look into the tree canopy, I think, LiDAR is a good option.

So we can actually using LiDAR to predict the size of the tree and then the canopy density, everything like that.

So, LiDAR would be doing very well on that aspect.

However, if we want to look into the crop itself, for example, flower clusters or fruit numbers, fruit counts on a tree, then the LiDAR sensor potentially has a little bit of limitation there to see the flower kind of density to kind of classify the flowers and fruits on the tree, the crop load, for example, right?

So, but that actually, I think, is a good point to integrate maybe the camera sensor and LiDAR sensor together to operate this together.

"And similar what going on, the ground-based specialty crops." The ground-based sprayer that what I showed there right now doesn't have any sensors on top of that, so but you can presetting, for example, the flow rate of each nozzle.

You can preset the angle of the spray.

So those can be preset, and once it preset, and then the sprayer just driving through the orchards on the paths that the map gives the sprayers to go.

So, but like I said, LiDAR, it's very kind of, it's very important to integrate these two technologies together.

So, if we have a LiDAR sensor or a camera sensor on top of this robotic ground sprayer, that actually can both do the automation but also can do precise spray.

The next question is, "Are there any attempts to do any harvest foods quality and quantity mapping?" Yes, you see that last, I kind of introduced a robotic harvesting project.

So, in that project, we are actually doing this work.

We're trying, firstly, to estimate the crop load, kinda the number of fruits on the tree, but also, we would like to look into the quality of fruits.

For some of the fruits on the tree, if they're on the low side of quality, we won't be able, we're not going to harvest those fruits.

So only kind of doing some selective harvesting.

So those kind of like the quantity and quality map would be very helpful for robotic selective harvesting.

And here, "Sorry, I'm talk to actually using laser, like laser weeder, for destroying the bud." Oh, oh, okay.

Sorry, I misunderstood your question.

So here, the question is like can we use laser point to destroy the buds, flower buds for thinning purpose.

That actually has potential.

I would think there's already some research has been done in the past to using laser, laser kind of heat to kill the buds.

I do believe there's some study has been done.

But it's in terms of practically using this because it's laser is very, very kind of small points, so it need to be very targeted, and then the flower bud's also very small.

So this accuracy kind of, it need to be very high precision, so otherwise, it going to miss the buds, and then it will damage other part of the tree or something else.

So, I think it's something that has potential but, at this moment, I don't think anybody has been doing this using a robotic system to do this study.

Some previous study has been done for like laser killing the buds.

That, I think that's...

I didn't read too much close to that research, but I think that's a potential kind of practical, potential solution for that, but how that can be integrated into a robotic system need to be investigated more in the future.

I think I answered all the question in the queue here.

- You did a fantastic job getting through all those questions.

Thank you very much.

- Thank you so much.

- And again, thank you, Dr. He, for taking the time to share some of your work.

I know you have so many things going on, and there's such a diversity of your work.

It's really phenomenal to see what you have going on.

I'm gonna share a screen real quick here, and just we'll wrap up, so.

First of all, immediately following the webinar or shortly after the webinar, you'll be receiving an evaluation.

Please take the time to complete the evaluation.

It does help us in planning future evaluations.

The whole point of this work is to build that collaboration and to be able to do things that have an opportunity to make a positive difference for growers and producers all across the United States and here in Pennsylvania, especially.

So please take a moment to let us know your thoughts 'cause it does help us in planning the future.

Also, just by way of a quick plug, our next webinar will be Harvesting Innovation: Connecting Geospatial Intelligence, High-Speed Internet, and Precision Agriculture, and we have just some fantastic people coming in to present.

Harry Crissy, who's an extension educator and geospatial intelligence analyst, he developed a national broadband map that's really been valuable.

Actually, he did some work for the FCC that found, on average, 96.1% of all US crops grown in the United States are within 10 miles of fiber optic cable, which, to me, that means that precision agriculture and connected information is really possible for farms all across the United States.

So if you'd like to join us, that will be on April 22nd at 1:00 PM.

If you participated in this webinar, you should get a notification about that webinar and the opportunity to register.

Also, if you'd like to continue the conversation and be part of our interest group, we do have a Ag Tech Interest Group.

All you gotta do is scan that QR code, and it'll take you to, basically, it's a mailing list sign-up.

We don't sell your information.

It's actually, we're not allowed to do that.

We don't give your information away to anybody.

We just use it internally, so your information will not show up anywhere else except for emails occasionally from us about ag tech-related topics.

So also, if you're looking for additional information from Penn State, the new Technologies for Agriculture and Living Systems initiative, which was started just last year.

Dr. Paul Heinemann is the director of this particular initiative, and Dr. Heinemann just has a tremendously long and great career in developing new technologies, and we're really pleased that he's leading this particular initiative on behalf of the university or on behalf of the college.

And again, there's more information available looking at that Technologies for Ag and Living Systems.

So, with that, I'm gonna say thank you, everybody, and we hope you enjoyed the presentation today.

And we will be continuing to do our Harvesting Innovation Webinar Series throughout the year.

I think we have, I don't know, eight more scheduled, a lot, so, and we'll be adding an IoT for Ag webinar series here soon.

So keep your eyes open for those extension mailings on how to sign up.

And if you ever have any questions, you can always reach out to Dana or myself, and we'd be happy to direct you or even connect you to other faculty members here at the university.

Thanks, everybody.

We hope you have a wonderful day, and we really appreciate you taking the time to join us.

Have a great day, everybody.

- Thank you, everyone.