Grapes of Math

George Fox computer science and engineering students are developing an autonomous robot that could drastically change the way vineyard owners predict their harvest By Kimberly Felton

They’re calling it the Vitibot. The name isn’t official, but a student designed a sticker with the name and it stuck – a clever combination of the term “viticulture” (the science of grapes) with “robot.”

One of the longest-running senior design capstone projects for George Fox computer science and engineering students, the Vitibot moves closer each year to the goal – a product the wine industry surrounding the university is eager to see: a rover that can autonomously traverse a vineyard, gathering image data, to accurately predict the grape yield months before harvest.

A 10 to 15% margin of error is considered a good estimate by most wine growers. Last year, the Vitibot predicted a yield with a 2 to 3% margin of error.

A Roomba for Grapes

Every year, wine growers closely monitor their vines to predict grape yield. Weather, soil, vine disease and pests like birds and deer affect that yield – but regardless of these elements, vineyard owners need to know if they can meet their commitments to buyers, and if they will end up with excess grapes. In the risky business of growing crops, their success hinges on an accurate prediction.

Iterations of the Vitibot have been around longer than most senior design projects, which usually begin in the fall semester and end in the spring.

“I started here in 2015, and already then it was a thing,” says computer science professor Brian Snider.

Bob Harder, dean of the College of Engineering, had brainstormed with local entrepreneurs and vineyard and winery owners, developing the technology for a few years before Snider’s arrival at the university.

“From the beginning, it was first and foremost a mechanical engineering project,” Snider says. “But Bob knew there was a need for machine learning, and that’s where I became involved.”

Two teams of seniors – new teams each year – worked on parallel endeavors with the Vitibot. The hardware, or data collection, team made up of mechanical and electrical engineering students designed and built the rover. The software, or data analysis, team of computer science and information systems students worked on the machine learning and autonomous navigation – the artificial intelligence. Finally, this year, the two teams merged into one, stitching together the mechanics and sensor data. “And that’s when we got the thing to drive by itself,” Snider says.

Snider and other faculty in the College of Engineering envision a rover they can sell, that has a package of sensors, and can be maneuvered by remote control or drive autonomously (currently the Vitibot uses LiDAR sensors for navigation and GoPro cameras for data collection).

“It’s like a Roomba, almost, but for your crops,” Snider says. “You let it navigate the vineyard rows by itself and collect data as it goes.”

From left to right, students Trey Moen and Jared Perry, along with professor Brian Snider, test the Vitibot with vineyard owners Jeanne and David Beck.

A Community Effort

To create a rover that could take digital images, feed those images into a machine learning model, and use the trained model to predict yield, the teams needed a testing ground. David and Jeanne Beck, owners of Crawford Beck Vineyard, willingly offered their vineyard. Scientists, researchers and educators themselves, they’re equally interested and invested in the technology that can assist their vineyard, and the process of allowing students to learn.

The Vitibot’s accuracy in grape yield prediction is dependent on two things: weekly photos of the vines and machine learning built into the rover’s “brain.”

The past three summers, a student or faculty advisor walked the rows at Crawford Beck Vineyard, carrying an eight-foot hiking stick with three GoPro cameras strapped on. Each week, from July until harvest in mid-September, they took a photo of each vine. Eighty-three plants per row, 21 rows, 10,000 images every week.

“We have an algorithm, a piece of code, that accepts as input the pictures,” Snider says. “And we have code that learns relevant numerical features from the images. That information goes in; we call it training data. Then, because we’re doing supervised learning, we tell it, ‘Given these images, here’s the correct answer you should predict.’”

The “correct answer” given to the algorithm is the grape weight at harvest. After uploading images all summer, the actual weight is added to the algorithm at the end of the season. This is where the Becks’ contribution looms large.

In most vineyards at harvest, workers run the rows, dropping clusters into buckets, emptying buckets into bins, getting credit for each bucket they pick. It’s competitive and it’s fast. It saves labor costs.

But to validate the rover’s software, the premium no longer is on time; it is on harvest weight accuracy, not just by the vineyard, but also by the winery – in this case, Winderlea.

“They’re willing to weigh all those bins of harvested grapes and give us the data,” David says. “That’s what ties it all together and validates the success of the software.”

The process is time-consuming but absolutely necessary in order to develop an accurate model. The Becks and Winderlea are willing.

How the Vitibot works its way from images to the final weight of the grapes, Snider cannot explain. “You don’t necessarily know what features it’s latching onto. Is it counting grapes? How does it know? But that’s the thing; we don’t need to know. We’re going to let this algorithm learn what is relevant.” The numbers prove that it is working.

“Their software is better than our actual measurements,” David says. “We count clusters and we weigh clusters, and we gross that up by the total number of vines. We do an informed calculation, but we’re making assumptions about how evenly the fruit is spread.”

Like most wine growers, Jeanne handles grape yield prediction herself. It is tedious, time-consuming and not as accurate as she would prefer.

“As for the margin of error that winemakers want, the answer is zero!” she says. “That not being possible, we strive for both labor conservation and maximal accuracy. Given this inherent inaccuracy, some growers opt for optical assessments, walking the rows and estimating by eye. This is not satisfactory, so we constantly search for better methods.

“And that is the beauty of the work the George Fox students have done. Their estimation was within 98% of the harvest amount. Mine was only 72% of the harvest amount. Add to that the labor savings and you can see how impressive this is.”

The Vitibot isn’t ready for market yet, but the software has consistently produced accurate yield predictions, and this year was able to drive autonomously. “Now that we’ve broken through that wall, the next step is to get it out to the vineyard every week and train it to drive in an actual, real-life environment,” Snider says. “And we’re pretty optimistic that next year we’ll get there.”