{"id":10900,"date":"2021-04-30T13:41:42","date_gmt":"2021-04-30T11:41:42","guid":{"rendered":"https:\/\/advancedfleetmanagementconsulting.com\/eng\/?p=10900"},"modified":"2021-04-30T13:41:42","modified_gmt":"2021-04-30T11:41:42","slug":"data-storage","status":"publish","type":"post","link":"https:\/\/advancedfleetmanagementconsulting.com\/eng\/2021\/04\/30\/data-storage\/","title":{"rendered":"Data Storage Trying to Keep Up with a Speeding Car"},"content":{"rendered":"

Development and testing of autonomous vehicle technology is data intensive, requiring solutions for the management and storage and of rapidly expanding volumes of data.<\/span><\/p>\n

The primary reason for this explosion is the collection of video and LiDAR data, which has to be high-resolution. Naturally, the more sensors one has on the AV, the more data the AV would generate. An AV sensor suite may comprise of eight cameras, two LiDAR sensors and two radars sensors or it may comprise of just one LiDAR and a couple of cameras.<\/span><\/p>\n

Depending on the operating conditions, for example the operational design domain (ODD), of the automated driving system (ADS), the manufacturers\u2019 choice of sensor suite may differ substantially. For example, the resolution of data required for a low-speed ADS application in a business park (constrained ODD) may be different to a high-speed application on highways.<\/span><\/p>\n

\u201cVehicles are collecting data 24\/7 as they create a 3D world via LiDAR and video and there is no easy way to compress data,\u201d explained Ken Obuszewski, global general manager of NetApp\u2019s automotive vertical. \u201cTraditional methods of data reduction don\u2019t work in this context.\u201d He said when it comes to managing this data, it is critical to have rich metadata about the data you have captured in order to optimize the processing of the data, while data tiering and archiving to the cloud are necessary to store and retain massive amounts of data.<\/span><\/p>\n

Data management methods such as tiering and archiving allow for proper insight into what data gets stored, and what data gets discarded. For storing the data, Obuszewski said that because these huge volumes of data are generated in multiple locations, the challenge is how to provide and access that data where and when you need it. In short \u2013 flexibility and locality matters.<\/span><\/p>\n

\u201cVirtual desktop infrastructure is important, sometimes it\u2019s easier to bring the engineer to the data,\u201d he said. \u201cThe flexibility and delivery of data is made more efficient by the ability to create a data fabric across the data pipeline, a way to store and move data from the edge to the core and cloud and back again.\u201d Obuszewski said some of the \u201cheavy lifting\u201d, such as AI processing needs, should be centralized for efficient data movement, noting data must be moved in a secure and efficient manner.<\/span><\/p>\n

In the next 18-24 months, he sees more data processing being done at the edge, capturing event-based data. He also predicted an evolution towards more efficient use of the data, such as being able to identify and keep only what they feel is valuable and using AI at the edge to both label and identify event-based data to reduce the amount of data being retained, moved, and managed. This includes data intelligence strategies that use AI at the edge and being more targeted and determine what is being captured. \u201cThese concepts will make the data more valuable and increase the quality of the data you train against,\u201d he said.<\/span><\/p>\n

Obuszewski noted cloud technologies would also continue to play a major role, with data being centralized within the cloud, providing compute resources and state-of-the-art tools and applications in the development environment. \u201cThe cloud is the environment of choice for the data scientist, because it provides flexibility and availability, as well as long term data and cold data storage, offering cost efficiencies, for example, and legal and compliance needs,\u201d he said. \u201cAutomakers are looking for ways to efficiently store this explosion of cold data. The cloud provides that option.\u201d<\/span><\/p>\n

Dr Siddartha Khastgir, head of verification and validation, intelligent vehicles at WMG, University of Warwick, pointed out that depending on the operating conditions, i.e. ODD of the ADS, the manufacturers\u2019 choice of sensor suite may differ substantially. Khastgir explained the crucial aspect of AV testing is not in understanding the quantity of data but rather it resides in understanding \u201cwhat data\u201d is useful. \u201cData storage and management is about storing the \u2018right\u2019 data and the challenges are associated within the process of defining the right data and subsequently extracting it,\u201d he said. \u201cThus, there is now a shift in focus from storing \u2018all data\u2019 to storing the \u2018right data\u2019.\u201d<\/span><\/p>\n

Michael Erich Hammer, who leaders the Big Data intelligence team at powertrain development, simulation and testing firm AVL, said at the moment, it is still unclear what the main strategy will be on handling and storing such large amounts of data. \u201cThe costs for hot storage of data in the Petabyte range is huge,\u201d he noted. \u201cTherefore, a continuous trade-off between scalability, performance and costs between cloud and on-premise storage has to be expected.\u201d<\/span><\/p>\n

He noted transfers of large amount of data (>10Gbit\/sec created in the vehicle) will not be possible via the network for quite some time. \u201cThe workflow contains shipment of physical storage devices from the vehicle to back-office environments,\u201d he explained. \u201cThis \u2018long loop\u2019 from data creation to information extraction, with potential for delays lasting more than a week, needs to be supported by a \u2018short loop\u2019 for live fleet monitoring during operation of the vehicles on road.\u201d<\/span><\/p>\n

Like Obuszewski he said when it comes to scalability in processing information out of the data, there is \u201cno doubt\u201d about the advantages of cloud-based services. \u201cWe expect to see hybrid solutions where workloads are executed on premise and in cloud environments,\u201d he said. \u201cOne important point is to first reduce the data necessary to store.\u201d<\/span><\/p>\n

This means extracting relevant driving situations, or so-called \u201cscenarios\u201d, which are key to validate the AVs. \u201cFor example, highly dynamic scenarios with more traffic participants and also vulnerable road users involved are considered more interesting than just simple, constant driving scenarios,\u201d Hammer said. \u201cThis includes scenarios where an unexpected behavior of the environment or other traffic participants as well as the AV occurs, would be of great interest.\u201d<\/span><\/p>\n

Khastgir pointed out that depending on the system architecture for the AV application, the cloud could play an important role in data storage and management, and like Hammer, said the challenge currently lies in being able to get the data off the AV and get it stored in the cloud quick enough. \u201cHigh speed connectivity solutions like 5G may offer a solution to this challenge, along with data compression methods,\u201d he said. \u201cHowever, coverage and signal strength of 5G connectivity may pose a challenge for such an approach.\u201d<\/span><\/p>\n

He added that standards like ASAM OpenLabel, which are focusing on ways to label sensor data, are a big step towards getting the industry to speak the same language for raw data. There are some activities which are taking place (or have just taken place) like Data Storage System for Automated Driving (DSSAD) at UNECE, ASAM OpenLabel and the BSI PAS 1882 to provide some guidance on this aspect.<\/span><\/p>\n

\u201cHowever, a more detailed description may be required to help manufacturers overcome data storage challenges,\u201d Khastgir said. \u201cI do see more efforts on both technology and standardization fronts happening for off-board storage.\u201d<\/span><\/p>\n

 <\/p>\n

by Nathan Eddy<\/a><\/p>\n

Source: https:\/\/www.tu-auto.com<\/a><\/span><\/p>\n

CUT COTS OF THE FLEET WITH OUR AUDIT PROGRAM<\/strong><\/a><\/h3>\n

\"\"<\/a><\/p>\n

The audit is a key tool to know the overall status and provide the analysis, the assessment, the advice, the suggestions and the actions to take in order to cut costs and increase the efficiency and efficacy of the fleet. We propose the following fleet management audit.<\/p>\n

FLEET MANAGEMENT AUDIT<\/strong><\/a><\/h3>\n","protected":false},"excerpt":{"rendered":"

Development and testing of autonomous vehicle technology is data intensive, requiring solutions for the management and storage and of rapidly expanding volumes of data. The primary reason for this explosion is the collection of video and LiDAR data, which has to be high-resolution. Naturally, the more sensors one has on the AV, the more data…<\/p>\n","protected":false},"author":3,"featured_media":10901,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[18],"tags":[39],"_links":{"self":[{"href":"https:\/\/advancedfleetmanagementconsulting.com\/eng\/wp-json\/wp\/v2\/posts\/10900"}],"collection":[{"href":"https:\/\/advancedfleetmanagementconsulting.com\/eng\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/advancedfleetmanagementconsulting.com\/eng\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/advancedfleetmanagementconsulting.com\/eng\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/advancedfleetmanagementconsulting.com\/eng\/wp-json\/wp\/v2\/comments?post=10900"}],"version-history":[{"count":1,"href":"https:\/\/advancedfleetmanagementconsulting.com\/eng\/wp-json\/wp\/v2\/posts\/10900\/revisions"}],"predecessor-version":[{"id":10902,"href":"https:\/\/advancedfleetmanagementconsulting.com\/eng\/wp-json\/wp\/v2\/posts\/10900\/revisions\/10902"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/advancedfleetmanagementconsulting.com\/eng\/wp-json\/wp\/v2\/media\/10901"}],"wp:attachment":[{"href":"https:\/\/advancedfleetmanagementconsulting.com\/eng\/wp-json\/wp\/v2\/media?parent=10900"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/advancedfleetmanagementconsulting.com\/eng\/wp-json\/wp\/v2\/categories?post=10900"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/advancedfleetmanagementconsulting.com\/eng\/wp-json\/wp\/v2\/tags?post=10900"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}