Infrastructure free localisation in dynamic port areas
Optimized navigation needs precise localization. Localization can be based on various sensor modalities. Each of them has its pros and cons. Modern ports are filled with infrastructure to support the localization process in the hunt for the perfect solution for all mobile machines operating in the area. Unfortunately, the case is not trivial, and the port operators are constantly struggling to find balance between manned and unmanned operations – not solely, but partly, due to the difficulties related to autonomous navigation. GIM Robotics has been tackling this important problem with some of our friends.
Containers need proper machines
Reach Stackers, Terminal Tractors, Flatbeds, and Forklift Trucks are all sizable machines, which scream for respect when you stand next to them in an open yard. After all, they are developed to move full size containers effectively around the port area and beyond. However, as an ocean container is just a standard piece of an extremely large 3D, or should we say 4D for its dynamic nature’s sake, puzzle, these large machines are somewhat dwarfed by the sheer dimensions of the huge walls and blocks built from tens or hundreds of containers placed next and on top of each other’s according to complex plans created by the all-mighty port level automation system. This feeling would be further strengthened if you would manage to see those machines next to the absolute mammoths of the port areas, i.e., next to the Ship-to-Shore Cranes, Rubber-Tyred Gantry Cranes, Stacking Cranes, and Straddle Carriers.
Size really matters
The smaller size of these machines becomes an issue, when you start to plan how to localize those machines in that highly constrained and dynamic environment, where GNSS availability is a real challenge, fixed poles/pylons/structures attached beacons/reflectors are not constantly visible and on ground placed magnetic beacon system is very rigid by nature and thus extremely costly to modify and expand when needed. RF-based systems have their perks, but they suffer from various issues like line-of-sight and environmental conditions related challenges, which in turn might require a considerable amount of hardware to secure the full coverage in all conditions.
Feature-based localisation
To address the above issues, the industry has, already for some time, been implementing various types of feature-based localization solutions. Point-cloud producing sensors have been used to create 2D or 3D models of the environment and the machines have been localized in that model, or on the map, if you like. Normally used sensors include LiDARs, radars and different types of camera solutions. Each of those sensor modalities have, again, their pros and cons, and the winning solution should balance between performance, prizing, and regulations as always. The optimal solution would obviously have several different sensors for the much-needed redundancy and robustness, but that will of course come with a higher price tag
Dynamic environment causes challenges
The feature-based localization methods are slowly turning into mainstream, but they struggle when the environment becomes highly dynamic. And unfortunately, a busy port area with huge material flow and many different mobile machines, with intentional and unintentional presence of humans, represents something which can only be considered as an extremely dynamic and challenging environment for robust and reliable feature-based localization. The massive container walls and blocks form a constantly changing world, where the mobile machine can’t trust that the wall, which it detected two hours ago, would still be there. It is easy to understand that this will cause serious challenges for any solution, which depends on the map of the environment
How to secure real-time maps
To mitigate the risk of losing accurate feature-based localization, some distinct actions must be taken to address these challenges, i.e., to make sure that the map is up to date all the time. There are several ways to achieve this. None of them is the proverbial Silver Bullet, should there exist any. The solution should be lightweight enough to run onboard the mobile machine to avoid extensive communication related challenges, but still guarantee high enough quality needed for precise feature-based localization.
The decision which must be made at this point of implementation is to decide whether you are happy with non-optimal solutions for a big part of the operation or should you aim for the optimal performance. If you are happy with non-optimal performance, you can use the Real-Time Simultaneous Localisation and Mapping (RT-SLAM) approach. It basically means that, even though you have the map of the environment, you are constantly running the mapping phase to make sure the map is up to date
SLAM has its issues
However, due to the innate properties of the SLAM, the newly acquired map is not perfect, or in many cases not even very good, until the uncertainties are reduced considerably when the loop closure takes place. In other words, while you are creating the map, it is not so good, until some type of loop closure takes place. Classically, this happens when the machine comes to a place where it has been before and is capable of detecting that. There are other ways to provide a kind of loop closure, like using special locations, which can be recognised as the machine arrives at one of those. The Points of Interest, or maybe more precisely, Areas of Interest, will have more or less the same effect as previously visited and recognized locations, i.e., they help reduce the uncertainty to acceptable levels.
An option for SLAM
Another way to secure the quality of the map is to use the existing map as long as possible, while continuously monitoring the changes and updating only the relevant parts to the existing map. This option prevents the inherent rise in the uncertainties related to the SLAM cycle, but on the other hand, one cannot use available well-matured SLAM algorithms and novel solutions for the task are needed. For more detailed presentation, see our Blog post.
A priori map
If we dare to fantasize for a second, that the operation of a busy port would somehow be jump-started periodically from a short resting period, it would be natural that the initial operational map would be based on the fixed topological map and on the information of the latest poses (position and orientation) of the containers and the machines. This map would be coming from the Port level automation system. That information would be then converted into the selected map format, whatever that might be. When the new shift starts, the map would have the best possible information available.
Seeing the full picture
The inherent problem related to large ports is their colossal size. Even though the active machine fleet would be extensive, the uncertainties related to the map, created solely based on information coming from the machines, are in any case extensive. There will be areas which have not been visited recently with the smaller machines and are thus lacking the accurate map data provided by their point-cloud producing sensors. Container corridor configurations are being changed partly by the big machines. The Port level automation system has that information, although with less precision. That information will be transferred immediately and automatically processed to the right format and included in the full-scale map available for the smaller machines.
The future is fantastic
The Kalmar AutoStrad™, the industry-leading Automated Straddle Carrier solution has been one of the success stories in this field. (As a side note it is good to mention here, that GIM Robotics has, unfortunately, nothing to do with it.) Autostrad, and the other pioneering solutions, are operating in constrained environments with restricted machine fleets. Currently, there is no real full-scale solution, which would include all, or at least the majority, of typical mobile machine types, which would be operating autonomously. Obviously, there are many reasons for that, but one important thing, which is slowing the introduction of a full system, is related to laws, rules and regulation specifying man-machine coexistence on port areas. Nowadays, autonomous operations are restricted to special constrained areas, which are in normal operation mode free of humans. In the future, that is bound to change. As a consequence, more machine types will be robotized and put to work together in areas where human presence is still very much regulated, but not totally forbidden. Once we get to that point, the above text will apply fully. We know that all the main machine types are already more or less capable of autonomous operations and we know that the most sophisticated full port level automation systems have the needed functionalities. It is just a question of when the legislation will enable the full-scale deployments. Once we get close to that point, it is pretty safe to claim that the needed R&D projects and the following productization processes will get thumbs up everywhere
When that happens, GIM Robotics will be there to serve.
Special thanks for the great video (see our Linkedin page) and smooth cooperation are due to our friends at Mevea Ltd.
