John Deere moves further in the field of autonomy

The fully autonomous production system for row crops that John Deere has targeted to deliver by 2030 represents the ultimate level of “sense and act” autonomy. (Photo: John Deere)

Since its Precision Ag Group was started in 1993, John Deere has steadily increased the level of automation in its products, edging further and further forward along the path to full autonomy. It’s now closer than ever to that goal, thanks to technologies being applied across multiple solutions within its Precision Ag portfolio.

“Big picture, we started this journey because of the need that these products can fill for farmers,” said Jorge Heraud, vice president of automation & autonomy, John Deere. “These products can really help farmers take productivity, sustainability, profitability to the next level.

“We have products both in… the automation side and on the autonomy side… A lot of them use similar underlying technologies, but the way that they work is a little bit different.”

“The idea is we’re finding ways with the new cutting-edge sensors, cameras, computer vision, machine learning to make the machines just do more,” said John Deere’s Jorge Heraud. (Photo: KHL Staff)

On the automation side are a collection of products, such as the See & Spray family of targeted spraying solutions and Combine Advisor for automatic separation of grain from chaff. These products use cameras and fast processors to tune the machine in the best possible way, making the farmer more profitable by saving chemicals, seed and collecting more grain, said John Deere.

Each of these technologies has helped further the company’s objective to develop a fully autonomous tractor and tillage system – a solution first teased at the Consumer Electronics Show (CES) in early 2022. Retrofit autonomy kits mounted on Deere’s 8R tractor and 2430 chisel plow have now been in customer field tests since last fall.

“As we talk about increasing levels of automation leading to ultimate autonomy, we’ve been preparing for the autonomous tractor solution… since we started with AutoTrac back in 2002,” Matt Olson, Precision Ag product marketing manager, John Deere, said of the company’s field guidance system. “We started in 2002 making customers more aware of the value of precision and how overlap control can really help not only increase the productivity per day, but also decrease the costs… relative to the inputs of their operation.

“Over time, we just continued to increase the level of automation solutions.”

Technology commonality

John Deere has seven new automation products currently in development, including ExactShot for precise fertilizer placement during planting; Furrow Vision for direct view of seeds as they are planted in furrows; and, on the construction side, Smart Detect for human and object detection behind wheel loaders.

“One of the cool things is that we use a lot of the same techniques across all the products,” Heraud said. “We’ve developed expertise and commonality in cameras, processing units and, in particular, our products use computer vision tied with machine learning.”

Automation starts with rugged John Deere-designed, high-resolution cameras that serve as the primary sensors on the equipment. Working individually or in pairs depending on the application, their high dynamic range functionality enables them to capture multiple images in rapid succession using different exposures.

Rugged John Deere-designed, high-resolution cameras serve as the primary sensors around the machine. (Photo: KHL Staff)

“One of the things we’ve encountered is you have a camera [mounted on] a boom, and sometimes the boom casts a shadow, sometimes it’s in bright sunlight,” Heraud noted. “If you were to take a regular picture, it would be too dark or too bright. But with this camera that is high dynamic range, we’re able to see both of them.”

The rapid processing units and even the software used to collect data and “train” machines are largely the same across automated products. “There’s little differences here and there because of where cameras are mounted and what exactly we are trying to detect,” Heraud said.

“The idea is we’re finding ways with the new cutting-edge sensors, cameras, computer vision, machine learning to make the machines just do more. And in many cases, it’s not just a little bit more, but it’s huge amounts more.”

A case in point, he said, is See & Spray, which is able to “see” and then target each weed in a field, rather than use a broad spread application. The result is a third of the herbicide being applied compared to a standard sprayer. Even with a limited production build of the system in 2022, this translated to a 2.2 million gal. savings in herbicide. “So, you can imagine that next year is going to be 10 times more. It’s quite effective,” Heraud said.

Sense and act objective

What John Deere is striving toward is “sense and act autonomy,” said Olson. “This is where we really get into machine learning, artificial intelligence, fully autonomous machines. It’s all about a proliferation of sensors, cameras and controllers that are giving data to these systems and a decision is going to be made.”

Sense and act is already at work in the company’s automation solutions. For example, when Combine Advisor senses the loss of grain, it makes an adjustment to ensure the grain is retained. “When we look at it from an autonomous vehicle perspective, the machine is going through the field, senses an object [and is going to] steer around it or make some kind of correction,” said Olson.

John Deere’s new Precision Ag platform includes the StarFire 7000 Universal receiver. (Photo: KHL Staff)

The fully autonomous production system for row crops that John Deere has targeted to deliver by 2030 as part of its LEAP Ambitions represents the ultimate level of sense and act autonomy, Olson noted. To get there, the company introduced a brand-new Precision Ag platform at the Commodity Classic in March, which includes new G5 Universal displays, a new JDLink M modem and the StarFire 7000 Universal receiver.

“This full system is the system of Precision Ag for the future,” Olson said. “When we look at the new StarFire 7000 receiver and StarFire RTK (SF RTK), we now have satellite-based RTK corrections that allow sub-inch repeatable accuracy. When we look at autonomous machines, we need high precision, and we need that precision no matter where these machines are going.” He added that the StarFire 7000 is 73% faster than the StarFire 6000 it replaces.

The next piece is the G5 series of displays, which provide a 35% larger viewing area, three times faster processing speed and a high-definition screen, said Olson. “So, a lot of power there to be able to process the kind of knowledge that’s coming from… sensors and cameras and controllers to make higher levels of automation at the productivity and accuracy that customers expect.”

Olson calls modems the “unsung hero” serving as the “telematics pipeline” to and from the Operations Center platform used to manage both automation and autonomy products. The new JDLink M modem can be used on legacy equipment, mixed fleets and support vehicles, enabling farmers to essentially replicate their operation in the Operations Center platform, he said.

AI enables autonomy

In early April, John Deere demonstrated to the media for the first time a stock 8R tractor and 2430 chisel plow outfitted with an autonomy kit at its newest test farm, located roughly 40 miles from its Austin, Texas-based Innovation Hub. According to Maya Sripadam, senior product manager at Deere subsidiary Blue River Technology, the kit is built upon Deere’s Precision Ag stacks, including but not limited to:

• Six cameras to provide 360-degree visibility around the machine, plus premium lighting for nighttime visibility;
• High-speed computer processors to capture data;
• StarFire 7000 receivers on both the tractor and implement for GPS localization guidance;
• AutoTrac Full Field guidance, an expanded system being developed specifically for autonomy;
• And the Operation Center mobile app, which is used to control and monitor the machine.

By design, much of the technology will be familiar to John Deere customers. “The idea was that we’re going to add autonomy on top of this Precision ag stack that customers have already adopted,” Sripadam said.

“The way that the product works is it has cameras that see 360 degrees around the tractor and we use AutoTrac, we use GPS, to navigate as we normally do,” Heraud explained.

If the cameras spot an object in front of the machine, the tractor stops and sends the captured images to Spark.AI, a newly acquired John Deere company that uses a combination of human cognition and artificial intelligence to drive autonomy. Spark.AI technicians “triage” the situation to determine if the object is a real obstacle or a “false positive,” such as a shadow or weed. If a false positive, the tractor is told to continue forward.

“The idea is not to distract the farmer unless there’s a real obstacle and allow the machine to be very productive,” Heraud said.

When an object is identified as a real obstacle, the farmer is notified via the Operations Center mobile app on their phone and is able to view what the tractor is seeing. The farmer can then instruct the tractor to take a specific action or remain stopped as needed.

The use of Spark.Ai for object identification is an intermediate step toward full autonomy, with each detection event used for machine learning – training the system on what’s real and what isn’t. “The idea is that over time, [the system gets] better and better and we start getting less and less false positives, and the tractor will just react for the real obstacles only,” Heraud said. “At that point, we won’t have very many false positives, so the farmer can then triage.”

Confidence in the system’s ability to identify objects is essential to build trust in its ability to safely and effectively do its job. “What we’ve heard from customers is key features that you get through Operations Center mobile, like seeing that remote camera feed, seeing your coverage map, seeing your intelligence and being able to watch your tractor on the screen even when you’re not watching it in the field… really helps improve that trust,” said Joseph Liefer, senior product manager – autonomy, John Deere, “because you know that the machine is doing what you told it to do. And when you want it to stop and you want it to listen to you, we’re giving farmers control to manage their own equipment.”

John Deere demonstrated a stock 8R tractor and 2430 chisel plow outfitted with an autonomy kit. It also launched its autonomy prep option for the company’s large frame tractors. (Photo: KHL Staff)

Autonomy ready

John Deere recently launched its autonomy prep option for its large frame tractors (8 and 9 series tractors). The autonomy prep option bundles many of the known hardware components for customers that will want to run autonomously in the future. Other hardware will be needed in the future and John Deere is working to make autonomy capable with existing tractors.

The fully autonomous tillage system is still in the development stages, as John Deere continues to work with customers to identify their needs as well as understand the nuisances associated with different environments, changing field conditions, etc., that can impact its operation. In the meantime, customers continue to familiarize themselves with the technologies that will eventually drive its operation.

“StarFire receivers, our Command Centers that are already in the tractor, Operations Center mobile app, AutoTrac Turn Automation, Section Control – all of those features and that Precision Ag technology is available to farmers,” Liefer pointed out. “They can get comfortable with that technology today.”

AutoPath, an automation system used to help plan guided lines and field layout, is a perfect example. “As farmers adopt AutoPath, they’ll start to preview how autonomy is going to farm their fields when they choose to adopt that technology,” Liefer said. “All this technology builds on itself… Today, it’s multiple products that farmers use when they’re out doing the job… We’re building it into one seamless solution.”