Robot external and internal environment sensing by extraction of raw sensor data and their interpretation is the basic principle of robot perception. In the modular or end-to-end robot navigation approach, sensors play a critical role in capturing the environment or internal robot attributes for robot perception. Therefore, to achieve better perception, a wide range of different sensor modalities have been investigated by researchers. A sensor modality represents a sensor that inputs a particular form of energy and processes the signal using similar methods. Modalities include raw input types for sensors like sound, pressure, light (infrared, visible), or magnetic fields. Robot perception sensor modalities commonly include cameras (infrared, RGB or depth), LiDAR, radar, sonar, GNSS, IMU, and odometry sensors.
For more information, please visit our website.
The concept of robotic perception pertains to various robotics applications that utilise sensory information and modern deep learning methods. These applications include identifying objects, creating representations of the environment, comprehending scenes, detecting humans/pedestrians, recognising activities, categorising locations based on meaning, scene reconstruction, and others. Towards the goal of fully autonomous robot navigation, robust and accurate environmental perception is a necessity. Passive RGB cameras are relatively low cost and can capture rich environment details. However, their perception abilities are vulnerable to environmental illumination changes and occlusions. Therefore, sensor fusion methods are important to gain robust perception.
Active sensors emit energy to the environment and measure the return signal from the environment. There are several types of active-ranging sensors, such as reflectivity, ultrasonic, laser rangefinder (e.g., LiDAR), optical triangulation (1D), and structured light (2D). The commonly used LiDAR sensor is an active-ranging sensor that shows improved performance compared to ultrasonic TOF sensors for perception in autonomous navigation systems [ 46 ]. LiDAR imaging is one of the most studied topics in the optoelectronics field. The TOF measurement principle is used by LiDAR sensors for obtaining depth measurements in different environments. Rotating LiDAR imagers were the first type to successfully achieve acceptable performance using a rotating-mirror mechanical configuration with multiple stacked laser detectors [ 47 ]. New trends in LiDAR technology are in the development of low-cost, compact, solid-state commercial LiDAR sensors [ 48 49 ]. The three most widely used LiDAR techniques utilise pulsed, amplitude-modulated, and frequency-modulated laser beams. The most common commercially available method is to use a pulsed laser beam, which directly measures the time taken by the pulsed signal to return to the sensor. In such sensors, time measurements require resolutions in picoseconds (high-speed photon detectors). Therefore, the cost of pulsed LiDAR sensors is comparably higher than the other two methods for equivalent range and resolution [ 50 ]. The range of a pulsed LiDAR is limited by the signal-to-noise ratio (SNR) of the sensor. Pulsed LiDAR sensors are suitable in indoor and outdoor environments due to their instantaneous higher peak pulse contrast to ambient irradiance noise. However, Amplitude Modulated Continuous Wave (AMCW) LiDAR sensors with the same average signal power have lower continuous wave peaks and, hence, are vulnerable to solar irradiance. AMCW LiDAR sensors are popular in indoor robot applications due to low SNR. In general, there is no significant depth resolution difference between pulsed LiDAR and AMCW LiDAR, but pulsed LiDAR may outperform AMCW LiDAR accuracy at the same optical power levels because of higher SNR. Increasing the modulation frequency of AMCW LiDAR sensors improves depth resolution but reduces ambiguity distance. Thus, pulsed sensors have higher ranges compared to AMCW sensors. AMCW sensors also have slow frame rates relative to pulsed sensors and usually underperform in dynamic environments. Frequency Modulated Continuous Wave (FMCW) sensors have higher depth resolution in comparison to pulsed and AMCW LiDAR sensors. These sensors can measure the depth and velocity of targets simultaneously, a highly advantageous feature for the autonomous vehicle industry. These sensors can also avoid interference from other LiDAR sensors because of the frequency modulation. FMCW sensors, however, require accurate frequency sweeps to generate emitter signals, which is a challenging task. FMCW sensor technology has been in continuous development but has not yet established itself at the commercial level. A summary of these three LiDAR technologies is provided in Table 3
Omnidirectional cameras are utilised in robotic applications where more information is needed about surrounding environments. These cameras provide a wider-angle view than conventional cameras, which typically have a limited field of view. Due to the lens configuration of omnidirectional cameras, the obtained images have distortions. Therefore, these cameras require different mathematical models, like the unified projection model, to correct image distortions. A summary of these different camera types and their advantages and disadvantages is shown in Table 2
Event cameras are asynchronous sensors that use different visual information acquisition principles than the standard frame-based image sensors. This camera type samples light based on the changes in the scene dynamics (asynchronously measuring brightness per pixel). These cameras currently cost several thousands of dollars. Event cameras have advantages like very high temporal resolution, high dynamic range, low latency (in microseconds), and lower power consumption than standard cameras. However, this camera type is not suitable for the detection of static objects. The main burden is the requirement of new methods (algorithms and hardware, e.g., neuromorphic-based approaches or spiking neural networks) to process event camera outputs to acquire data and information because traditional image processing methods are not directly applicable [ 44 ].
Stereo cameras are inspired by human eyes and use two lenses and separate passive image sensors to obtain two perspectives of the same scenes. They have been used in indoor and outdoor SLAM applications. These camera types use the disparity of the two images to calculate the depth information. Stereo camera vision does not suffer from scale drift. Available popular stereo cameras are Bumblebee 2, Bumblebee XB3, Surveyor stereo vision system, Capella, Minoru 3D Webcam, Ensenso N10, ZED 2, and PCI nDepth vision system. Stereo camera power consumption is generally between 2 to 15 watts. The maximum range of stereo cameras varies from 5 to 40 m at different depth resolution values. The accuracy of these sensors varies from around a few millimetres to 5 cm at the maximum range [ 40 ]. The cost of these sensors varies from one hundred to several thousand Australian dollars. RGB-D camera sensors consist of monocular or stereo cameras coupled to infrared transmitters and receivers. Kinect camera from Microsoft is a relatively inexpensive RGB-D sensor that provides colour images and depth information of image pixels. The Kinect sensor is mainly used for indoor robot applications due to the saturation of infrared receivers in outdoor scenarios from sunlight [ 41 ]. The Kinect sensor has three major versions: Kinect 1, Kinect 2, and Azure Kinect (the latest version). Kinect 1 uses structured light for depth measurement, and the other models use Time-Of-Flight (TOF) as the depth measuring principle. The newest Kinect model, Azure Kinect, also generates substantial noise in outdoor bright light conditions with a practical measuring range below 1.5 m [ 42 ]. In general, RGB-D sensors utilise three depth measurement principles: structured light, TOF, and active infrared stereo techniques. The structured light RGB-D sensors underperform compared to TOF techniques in measuring the range of distant objects. The structured light technique is vulnerable to multi-device interference. The TOF methods suffer from multi-path interference and motion artefacts. The active infrared stereo principle has drawbacks due to common stereo matching problems such as occluded pixels and flying pixels near contour edges [ 43 ]. Active infrared stereo cameras are identified as an extension of passive stereo cameras. They offer more reliable performance in indoor and some outdoor scenarios. However, they require high computation capability. Table 1 shows the main depth measurement methods that are used in RGB-D sensors.
Vision is crucial for a robot to navigate in unknown environments. Robots are equipped with vision-based sensors like cameras to understand environmental information. Cameras are generally cheaper than LiDAR sensors and are a type of passive sensor (although some depth cameras use active sensing). The monocular configuration is highly used in standard RGB cameras, for example, the GoPro Hero camera series. They are compact passive sensors that consume low power, approximately 5 watts, depending on resolution and mode settings. Monocular SLAM has been explored in the research literature due to its simple hardware design [ 32 37 ]. However, the algorithms that are used for it are very complex because depth measurements cannot be directly retrieved from static monocular images. Monocular cameras also suffer from pose estimation problems [ 38 ]. The pose of the camera is obtained by referencing previous poses. Hence, errors in pose estimation propagate through the process, and this phenomenon is called scale drift [ 28 39 ].
Camera sensors are usually incorporated in vision applications by researchers to retrieve environmental information for the use of mobile robots. However, LiDAR sensors have shown more reliability in low-light environmental conditions than cameras and produce highly accurate depth measurements. LiDAR sensors come with 2D or 3D mapping capability [ 36 ], and these sensors can generate high-fidelity point clouds of outdoor environments. However, these unstructured point clouds tend to become increasingly sparse as the sensor range is increased.
Achieving a reliable real-time understanding of external 3D environments is very important for safe robot navigation. Visual perception using cameras is commonly employed in many mobile robotic systems. Camera images can be efficiently and often effectively processed by CNN-based deep learning architectures. However, relying on one sensor can lead to robustness challenges, particularly in applications like self-driving vehicles, autonomous field robots, etc. Therefore, different sensor modalities are often combined to achieve better reliability and robustness for autonomous systems. The fusion of LiDAR and camera data is one of the most investigated sensor fusion areas in the multimodal perception literature [ 51 ]. Camera and LiDAR fusion has been applied in various engineering fields and has shown better robustness than camera-only robot navigation approaches [ 52 ]. This fusion strategy is more effective and popular than other sensor fusion approaches such as radar-camera, LiDAR-radar, and LiDAR-thermal camera. Still, the technical challenges and cost of sensors and processing power requirements have constrained the application of these methods in more general human activities. Recent deep-learning algorithms have significantly improved the performance of camera-LiDAR fusion methods [ 53 ], with monocular and stereo cameras mainly used with LiDAR sensors to fuse images and point cloud data [ 54 ].
56,57,64,65,Deep learning-based LiDAR and camera sensor fusion methods have been applied in depth completion, object detection, and semantic and instance segmentation. Image and point cloud scene representations include volumetric, 2D multi-view projection, graph-based and point-based. In general, most of the early methods fuse LiDAR and camera image semantics using 2D CNNs. Many 2D-based networks project LiDAR points onto respective image planes to process feature maps through 2D convolutions [ 55 58 ]. Several works have used point cloud processing techniques such as PointNet [ 59 ] to extract features or 3D convolutions [ 60 ] to detect objects in volumetric representations [ 61 ]. Some other LiDAR and image fusion methods use 2D LiDAR representations for feature fusion and then cluster and segment 3D LiDAR points to generate 3D region proposals [ 62 ]. Voxel-based representations and multi-view camera-LiDAR fusion approaches are utilised to generate 3D proposals in object detection. State-of-the-art camera-LiDAR semantic segmentation methods employ feature fusion methods to obtain 2D and 3D voxel-based segmentation results. Multi-view approaches map RGB camera images onto the LiDAR Birds-Eye-View (BEV) plane to align respective features from the RGB image plane to the BEV plane [ 63 66 ], and several other methods propose to combine LiDAR BEV features with RGB image features directly [ 51 54 ]. These direct mapping methods use trained CNNs to align image features with LiDAR BEV features from different viewpoints.
Computer vision has been a rapidly growing field in the past decade, and the development of machine learning methods has only accelerated this. Recently, deep learning strategies have influenced the rapid advancement of various computer vision algorithms. Computer vision includes subtopics like object detection, depth estimation, semantic segmentation, instance segmentation, scene reconstruction, motion estimation, object tracking, scene understanding and end-to-end learning [ 41 ]. Computer vision methods have been applied to a great extent in emerging autonomous navigation applications. However, these vision techniques may be less effective in previously unseen or complex environments and highly rely on the trained domain. Therefore, continuous improvements are being made towards the development of fully autonomous systems. Several state-of-the-art benchmarking datasets have been utilised to compare the performance of different autonomous driving vision methods. KITTI [ 67 ], Waymo [ 68 ], A2D2 [ 69 ], nuScenes [ 70 ], and Cityscapes [ 71 ] are some examples of these state-of-the-art autonomous driving datasets.
Dense Depth Prediction
Dense depth completion is a technique that estimates dense depth images from sparse depth measurements. Achieving depth perception is of significant importance in many different engineering industries and research applications such as autonomous robotics, self-driving vehicles, augmented reality, and 3D map construction. LiDAR sensors, monocular cameras, stereo cameras, and RGB-D cameras have been the most utilised in dense depth estimation applications, but these sensors have specific limitations in the estimation process. LiDAR sensors with high accuracies are costly to use in large-scale applications. Three main challenges have been identified in LiDAR depth completion [ 72 ]. The first relates to the fact that, in general, even expensive LiDAR sensors produce sparse measurements for distant objects. In addition, LiDAR points are irregularly spaced compared to monocular RGB images. Therefore, it is non-trivial to increase the depth prediction accuracy using the corresponding colour image. Secondly, there are difficulties in combining multiple sensor modalities. The third challenge is that depth estimation using deep learning-based approaches has limitations with regard to the availability of pixel-level ground truth depth labels for training networks. Another possible approach to depth estimation is by using stereo cameras. Stereo cameras, however, require accurate calibration, demand high computational requirements, and fail in featureless or uniformly patterned environments. RGB-D cameras are capable of depth sensing but have a limited measuring range and poor performance in outdoor environments. A technique called depth inpainting can be used as a depth completion method for structured light sensors like the Microsoft Kinect 1, and these sensors produce relatively dense depth measurements but are generally only usable in indoor environments. Dense depth estimation techniques generally up-sample sparse and irregular LiDAR depth measurements to dense and regular depth predictions. Depth completion methods, however, still have a variety of problems that need to be overcome. These challenges are primarily sensor-dependent, and solutions should overcome respective difficulties at the algorithm development stage.
Many state-of-the-art dense depth prediction networks combine relatively dense depth measurements or sparse depth maps with RGB images to assist the prediction process. In general, retrieving dense depth detail from relatively dense depth measurements is easier than from sparse depth maps. In relatively dense depth images, a higher percentage of pixels (typically over 80%) have observed depth values. Therefore, in similar scenarios, predicting dense depth is relatively less complex. However, in autonomous navigation applications, 3D LiDAR sensors account for only approximately 4% of pixels when depth measurements are mapped onto the camera image space, which creates challenges in generating reliable dense depth images [ 72 ].
Dense Depth from Monocular Camera and LiDAR Fusion
Depth estimation based solely on monocular images is not reliable or robust. Therefore, to address these monocular camera limitations, the LiDAR-monocular camera fusion-based depth estimation process has been proposed by researchers. Using monocular RGB images and sparse LiDAR depth maps, a residual network learning-based autoencoder decoder network was introduced by [ 73 ] to estimate dense maps. However, this method needs a ground truth depth image when retrieving sparse depth images during the network training process. In practice, obtaining such ground truth images is not simple or easily scalable [ 72 ]. To mitigate the requirement of a ground truth depth image, ref. [ 72 ] presented a self-supervised model-based network that only requires a monocular RGB image sequence and LiDAR sparse depth images in the network training step. This network consists of a deep regression model to identify a one-to-one transformation from a sparse LiDAR depth map to a dense map. This method achieved state-of-the-art performance on the KITTI dataset and considers the pixel-level depth estimation as a deep regression problem in machine learning. LiDAR sparse depth maps use per-pixel depth, and pixels without measured depth are set to zero. The proposed network follows an encoder-decoder architecture. The encoder has a sequence of convolutions, and the decoder has a set of transposed convolutions to up sample feature spatial resolutions. Convolved sparse depth data and colour images are concatenated into a single tensor and input to residual blocks of ResNet-34 [ 74 ]. The self-supervised training framework requires only colour/intensity images from monocular cameras and sparse depth images from LiDAR sensors. In the network training step, a separate RGB supervision signal is used (a nearby frame). LiDAR sparse depth can be used as a supervision signal. However, this framework requires a static environment to be able to warp the second RGB frame to the first one.
With this implementation, the root-mean-square of the depth prediction error reduces as a power function with the increment of the resolution of the LiDAR sensor (i.e., the number of scan lines). One of the limitations of this approach is that the observed environment needs to be stationary. If not, the network will not generate accurate results. Large moving objects and surfaces that have specular reflectance can cause the process of training the network to fail. These reasons reduce the applicability of this method in dynamic situations that are often present in outdoor environments. In addition, this network training process may become stuck in the local minimums of the photometric loss function due to improper network weight initialisation. This effect may result in output depth images that are not close enough to the ground truth because of the erroneous training process.
In [ 75 ], a real-time sparse-to-dense map is constructed by using a Convolutional Spatial Propagation Network (CSPN). The propagation process preserves LiDAR sparse depth input values in the final depth map. This network aims to extract the affinity matrix for respective images. The introduced method learns the affinity matrix by using a deep convolution neural network. The training process of the network model is achieved by incorporating a stochastic gradient descent optimiser. This network implementation showed memory paging cost as a dominant factor when larger images were fed into the PyTorch-based network. The CSPN network has shown good performance in real-time and thus is well-suited for applications such as robotics and autonomous driving. CSPN++ [ 76 ] is an improved version of the CSPN network with adaptively learning convolutional kernel sizes and numbers of iterations for propagation. The network training experiments were carried out using four NVIDIA Tesla P40 Graphic Processing Units (GPUs) on the KITTI dataset. This research shows that hyper-parameter learning from weighted assembling can lead to significant accuracy improvements, and weighted selection could reduce the computational resource with the same or better accuracy compared to the CSPN network.
Dense Depth from Stereo Camera and LiDAR Fusion
Estimating depth using stereo cameras provides more reliable results compared to monocular cameras. LiDAR sensors can produce depth measurements with improved accuracy over increased ranges and varying lighting conditions. The fusion of LiDAR and stereo camera sensors produces more accurate 3D mappings of environments than LiDAR-monocular camera depth completion. However, stereo cameras commonly have shorter detection ranges, and depth estimation becomes challenging in textureless environments and high occlusion scenarios. One of the significant works in LiDAR-stereo camera fusion is presented in [ 77 ]. This paper presents the first unsupervised LiDAR-stereo fusion network. The network does not require dense ground truth maps, and training is done in an end-to-end manner that shows a broad generalisation capability in various real-world scenarios. The sparsity of LiDAR depth measurements can vary in real-world applications. One advantage of the network proposed in this work is that it handles a high range of sparsity up to the point where the LiDAR sensor has no depth measurements. A feedback loop has been incorporated into the network to connect outputs with inputs to compensate for noisy LiDAR depth measurements and misalignments between the LiDAR and stereo sensors. This network is currently regarded as one of the state-of-the-art methods for LiDAR-stereo camera fusion.
Multimodal Object Detection
Reliable object detection is a vital part of the autonomous navigation of robots/vehicles. Object detection in autonomous navigation is described as identifying and locating various objects in an environment scene in the form of bounding boxes, including dynamic objects and static objects. Object detection may become difficult due to sensor accuracy, lighting conditions, occlusions, shadows, reflections, etc. One major challenge in object detection is occlusion, which consists of different types. The main occlusion types are self-occlusion, inter-object occlusion, and object-to-background occlusion [ 78 ]. Early image-based object detection algorithms commonly included two steps. The first stage was dividing the image into multiple smaller sections. Then, these sections were conveyed into an image classifier to identify whether the image section contained an object or not. If any object was detected in an image section, the respective portion of the original image was marked with the relevant object label. The sliding window approach is one way of achieving the above-mentioned first step [ 79 ].
A different set of algorithms uses a technique in contrast to the sliding window approach by grouping similar pixels of an image to form a region. These regions are then fed to a classifier to identify semantic classes (with grouping done using image segmentation methods). Further improved image segmentation can be achieved by using the selective search algorithm [ 80 ]. The selective search algorithm emphasises a hierarchical grouping-based segmentation algorithm. In this method, initially detected image regions are merged in a stepwise manner by selecting the most similar segments until the whole image represents a single region. These regions resulting from each step are added to the region proposals and fed to a classifier. The classifier performance depends on the used region proposal method. This object detection approach does not produce real-time performance suitable for autonomous navigation applications. However, advances such as SSPnet [ 81 ], Fast Regional-based Convolutional Neural Networks (R-CNNs) [ 82 ], and Faster R-CNN [ 83 ] were introduced to address this issue. The Faster R-CNN network generates a feature map by utilising the CNN layer output, and the region proposal generation is achieved by sliding a window (comprising three different aspect ratios and sizes) over the feature map. Each sliding window is mapped to a vector and fed to two parallel classification and regression networks. The classification network calculates the probability of region proposals containing objects, and the regression network indicates the coordinates of each of the proposals.
Object detection research has been mainly employed in the autonomous vehicle industry (for vehicle and pedestrian detection [ 84 ]) and mobile robotics. In contrast to camera-only object detection, sensor fusion has been implemented in different real-world applications to obtain more accurate and robust detection results. As previously discussed, LiDAR and camera sensor fusion are some of the most used and highest-performing sensor fusion methods. LiDAR and camera sensor fusion object detection approaches consist of two main techniques. These are the sequential and one-step models [ 54 ]. Sequential models use 2D proposal-based methods or direct 3D proposals to detect objects. In the sequential approach, 2D/3D regions are proposed in the first stage, and then 3D bounding box regression is done in the second stage. The 2D/3D region proposal stage incorporates fused image-point cloud regions that may contain objects. In the bounding box regression stage, feature extraction from region proposals and bounding box prediction is done. One-step models generate region proposals and achieve bounding box regression in parallel in a single step. The 2D proposal-based sequential approach uses 2D image semantics to generate a 2D proposal and point cloud processing methods to detect dynamic objects. This approach utilises already developed image processing models to identify 2D object proposals and then project these proposals to LiDAR 3D point cloud space for object detection.
Two approaches are mainly used to manipulate image-based 2D object proposals and irregular 3D LiDAR data. In the first method, image-based 2D bounding boxes are projected to the LiDAR 3D point cloud to implement 3D point cloud object detection algorithms. The second approach utilises point cloud projections on the 2D images and applies 2D semantic segmentation techniques to achieve point-wise semantic labels of the points within the semantic regions [ 54 ]. LiDAR-camera 2D sequential object detection methods include result, feature, and multi-level fusion strategies. These 2D proposal-based result-level fusion methods incorporate image object detection algorithms to retrieve 2D region proposals. These retrieved 2D object bounding boxes are then mapped onto 3D LiDAR point clouds. The enclosing points in frustum proposals are transferred into a point cloud-based 3D object detection algorithm [ 57 ]. The overall performance of this object detection approach depends on the modular behaviour of the 2D detection architecture. Sequential fusion may lose complementary data in LiDAR point clouds due to initial 2D image object proposal detection. Two-dimensional proposal-based feature-level fusion uses a mapping from 3D LiDAR point clouds onto the respective 2D images and employs image-based techniques for image feature extraction and object detection. One of these approaches appends per-point 2D semantic details as additional channels of LiDAR 3D point clouds and uses an existing LiDAR-based object detection method [ 85 ]. However, this approach is not optimal for identifying objects in a three-dimensional world because 3D details in the point clouds may be lost due to the projection.
Multi-level fusion combines 2D result-level fusion with feature-level fusion. This approach uses already available 2D object detectors and generates 2D bounding boxes. Then, points within these bounding boxes are detected. Subsequently, image and point cloud features are combined within the bounding boxes to estimate 3D objects. LiDAR and camera object detection using 3D proposal-based sequential models avoid 2D to 3D proposal transformations and directly generate 3D proposals from 2D or 3D data. This technique consists of two approaches, namely multi-view and voxel-based. MV3D [ 66 ] is a multi-view object detection network that uses LiDAR and camera data to predict the full 3D envelope of objects in the 3D environment. A deep fusion scheme was proposed to fuse features from multiple views in respective regions. The detection network comprises two networks: the 3D proposal network and the region-based fusion network. As the inputs, the LiDAR BEV, LiDAR front view and the RGB camera image are fed to the network. The LiDAR BEV is fed to the 3D proposal network to retrieve 3D box proposals. These proposals are used to extract features from the LiDAR front-view and camera RGB image inputs. Then, using these extracted features, the deep fusion network predicts object size, orientation, and location in the 3D space. The network was built on the 16-layer VGGnet [ 86 ], and the KITTI dataset was used for the training process.
One of the drawbacks of the multi-level fusion method is the loss of small objects in the detection stage due to feature map down-sampling. Combining image and point cloud feature maps by RoI (Regions of Interest)-pooling decreases the fine-grained spatial details. MVX-Net [ 65 ] introduces a method to fuse point cloud and image data voxel-wise or point-wise. Two-dimensional CNNs are used for the image feature extraction process, and a VoxelNet [ 87 ] based network detects 3D objects in the voxel representation. The input 3D LiDAR point cloud is mapped to the 2D image for image feature extraction in the point-wise fusion method, and then voxelisation and processing are done using VoxelNet. In voxel-wise fusion, the point cloud is firstly voxelised and then projected onto the image-based 2D feature representation to extract features. This sequential approach achieved state-of-the-art performance for 3D object detection at the time of its publication. Object detection utilising one-stage models performs object proposal retrieval and bounding box prediction in a single process. These detection models are suitable for real-time autonomous robot decision-making scenarios. State-of-the-art single-stage object detection methods, such as [ 88 ], simultaneously process depth images and RGB images to fuse points with image features, and then the generated feature map is used for bounding box prediction. The introduced method [ 88 ] utilises, two CNN-based networks to parallelly process point cloud and RGB front-view images. One CNN identifies LiDAR features, and the other CNN extracts RGB image features. Then, these RGB image features are mapped into the LiDAR range view. Finally, mapped RGB and LiDAR image features are concatenated and fed into LaserNet [ 89 ] for object detection and semantic segmentation. This network has been trained in an end-to-end manner. The network training was done for 300K iterations with a batch size of 128, distributed over 32 GPUs. The image fusion, object detection, and semantic segmentation process took 38 milliseconds on an Nvidia Titan Xp GPU.
Multimodal Semantic Segmentation
Scene semantic segmentation assigns a semantic category label to each pixel in a scene image, and it can be regarded as a refinement of object detection [ 90 ]. A scene can incorporate obstacles, free space, and living creatures (not necessarily limited to these categories). The complete semantic segmentation of images applies these semantic categories to all pixels across an image. Many recent computer vision methods rely on CNN architectures. These networks favour dense depth image data over sparse data. In general, LiDAR sensors produce irregular, sparse depth measurements. Ref. [ 73 ] introduced a technique to utilise LiDAR sparse data and RGB images to achieve depth completion and semantic segmentation in a 2D view. This network can work with sparse depth measurement densities as low as 0.8%, and at the time of its publication, this method showed state-of-the-art performance on the KITTI benchmark dataset. The base network of this prior work was adopted from NASNet [ 91 ], which has an encoder and decoder architecture. Using LiDAR and RGB image feature fusion, ref. [ 92 ] proposed a novel method to achieve 2D semantic segmentation in 2D images. This method introduced a self-supervised network that suits different object categories, geometric locations, and environmental contexts. This self-supervised network uses two sensor modality-specific encoder streams, which concatenate to a single intermediate encoder and then connect into a decoder to fuse the complementary features. The segmentation part is achieved with a network termed AdapNet++ [ 92 ] that consists of an encoder-decoder architecture. All these network models were implemented using the deep learning TensorFlow library. Another high-performing deep learning-based LiDAR-Camera 2D semantic segmentation method was presented in [ 93 ]. In this method, the generated 3D LiDAR point data is mapped to the 2D image and up-sampled to retrieve a 2D image set that consists of spatial information. Then, fully convolutional networks are used to segment the image using three approaches: signal level, feature, and cross-fusion. In the cross-fusion method, the network was designed to learn directly from the input data.
The techniques discussed up to now have been 2D semantic segmentation methods. In contrast to 2D methods, 3D semantic segmentation approaches provide a realistic 3D inference of environments. An early approach for a 3D scene semantic segmentation network is presented in [ 94 ] termed 3DMV. This method requires relatively dense depth scans along with RGB images, and it was developed to map indoor scenarios. Voxelised 3D geometries are built by using LiDAR depth scans. Two-dimensional feature maps are extracted from the RGB images using CNNs, and these image feature maps are mapped in a voxel-wise manner with the 3D grid geometry. This fused 3D geometry is then fed into 3D CNNs to obtain a per-voxel semantic prediction. The overall performance of the approach depends on the voxel resolution, and real-time processing is challenging for higher voxel resolutions. Therefore, this dense volumetric grid becomes impractical for high resolutions. The system was implemented using PyTorch and utilised 2D and 3D convolution layers already provided by the application programming interface. Semantic segmentation of point clouds is challenging for structureless and featureless regions [ 95 ]. A point-based 3D semantic segmentation framework has been introduced by [ 95 ]. This approach effectively optimises geometric construction and pixel-level features of outdoor scenes. The network projects features of detected RGB images into LiDAR space and learns 2D surface texture and 3D geometric attributes. These multi-viewpoint features are extracted by implementing a semantic segmentation network and then fused point-wise in the point cloud. Then, this point data is passed to a PointNet++ [ 96 ] based network to identify per-point semantic label predictions. A similar approach was followed by Multi-view Point-Net [ 97 ] to fuse RGB semantics with 3D geometry to obtain per LiDAR point semantic labels.
Instead of localised or point cloud representations, ref. [ 98 ] have used a high dimensional lattice representation for LiDAR and camera data processing. This representation reduces memory usage and computational cost by utilising bilateral convolutional layers. Then, these layers employ convolutions for unoccupied sections in the generated lattice representation. Firstly, the identified features of point clouds are mapped to a high-dimensional lattice, and then convolutions are used. Following this, CNNs are applied to detect image features from multi-view images, and these features are projected to the lattice representation to combine with three-dimensional lattice features. The generated lattice feature map was assessed by using 3D CNNs to obtain point-wise labels. A spatial-aware and hierarchical learning strategy has been incorporated to learn 3D features. The introduced network was capable of training in an end-to-end manner.
Multimodal Instance Segmentation
Instance segmentation identifies individual instances within a semantic category. It is considered a more advanced semantic segmentation method. This method not only provides per-pixel semantic categories but also distinguishes object instances, which is more advantageous for robot scene understanding. However, instance segmentation in autonomous navigation introduces more challenges than semantic segmentation. Instance segmentation approaches based on fused LiDAR-camera sensor data show proposal-based and proposal-free architectures. A voxel-wise 3D instance segmentation approach was introduced by [ 99 ] that consists of two-stage 3D CNN networks. A feature map was extracted from the low-resolution voxel grid by implementing 3D CNNs. Another feature map was obtained from the RGB multi-view images using 2D CNNs and projected onto the associated voxels in the volumetric grid to append with respective 3D geometry features. Then, object classes and 3D bounding boxes are predicted by feeding these fused features to a 3D CNN architecture. In the second phase, another 3D CNN estimates the per-pixel object instances using already identified features, object classes and bounding boxes.
These voxel-based segmentation methods are constrained by the voxel resolution and require increased computation capabilities with higher grid resolutions. The application of instance segmentation in LiDAR-camera fusion for real-time systems is challenging. Some research studies had limitations, such as the system developed in [ 100 ], which does not support dynamic environments. A proposal-free deep learning framework that jointly realises 3D semantic and instance segmentation is presented in [ 101 ]. This method performs 3D instance segmentation in the BEV of point clouds. However, this approach is less effective in identifying vertically oriented objects because of the BEV segmentation approach. This method first extracts a 2D semantic and instance map from a 2D BEV representation of the observed point cloud. Then, using the mean shift algorithm [ 102 ], and semantic features of the 2D BEV, instance segmentation is achieved by propagating 3D features onto the point cloud. It should be noted that the instance segmentation approaches discussed have been developed to segment 3D point clouds from static indoor environments, and these methods have not shown any segmentation capabilities in dynamic environments.
Overall, while significant progress has been made in the perception capabilities of autonomous robots, particularly in regard to object detection and scene segmentation, many of the existing approaches have only been tested in relatively structured indoor environments, and substantial additional work may be required to adapt these techniques to be used in unstructured outdoor environments. Nonetheless, some very useful research directions have been identified that have the potential to significantly advance this field.
Abel et al.
Abel, D., Barreto, A., Van Roy, B., Precup, D., van Hasselt, H. P., and Singh, S. ().
A definition of continual reinforcement learning.
Advances in Neural Information Processing Systems
,36
.Achiam et al.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. ().
Gpt-4 technical report..
Adeniji et al.
Adeniji, A., Xie, A., Sferrazza, C., Seo, Y., James, S., and Abbeel, P. ().
Language reward modulation for pretraining reinforcement learning.
ArXiv
,abs/.
.Ahn et al.
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Ho, D., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jang, E., Ruano, R. J., Jeffrey, K., Jesmonth, S., Joshi, N. J., Julian, R. C., Kalashnikov, D., Kuang, Y., Lee, K.-H., Levine, S., Lu, Y., Luu, L., Parada, C., Pastor, P., Quiambao, J., Rao, K., Rettinghouse, J., Reyes, D. M., Sermanet, P., Sievers, N., Tan, C., Toshev, A., Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Xu, S., and Yan, M. ().
Do as i can, not as i say: Grounding language in robotic affordances.
In
Conference on Robot Learning
.Ahn et al.
Ahn, M., Dwibedi, D., Finn, C., Arenas, M. G., Gopalakrishnan, K., Hausman, K., Ichter, B., Irpan, A., Joshi, N., Julian, R., et al. ().
Autort: Embodied foundation models for large scale orchestration of robotic agents..
Ahn et al.
Ahn, M., Zhu, H., Hartikainen, K., Ponte, H., Gupta, A., Levine, S., and Kumar, V. ().
Robel: Robotics benchmarks for learning with low-cost robots.
In
Conference on robot learning
, pp. . PMLR.Ajay et al.
Ajay, A., Han, S.-J., Du, Y., Li, S., Gupta, A., Jaakkola, T., Tenenbaum, J., Kaelbling, L. P., Srivastava, A., and Agrawal, P. ().
Compositional foundation models for hierarchical planning.
ArXiv
,abs/.
.Akkaya et al.
Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., et al. ().
Solving rubiks cube with a robot hand..
Alayrac et al.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. ().
Flamingo: a visual language model for few-shot learning.
Advances in neural information processing systems
,35
, .Anwar et al.
Anwar, U., Saparov, A., Rando, J., Paleka, D., Turpin, M., Hase, P., Lubana, E. S., Jenner, E., Casper, S., Sourbut, O., et al. ().
Foundational challenges in assuring alignment and safety of large language models..
Ardón et al.
Ardón, P., Pairet, È., Lohan, K. S., Ramamoorthy, S., and Petrick, R. ().
Affordances in robotic tasksa survey..
Arnab et al.
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., and Schmid, C. ().
Vivit: A video vision transformer..
Aytar et al.
Aytar, Y., Pfaff, T., Budden, D., Paine, T. L., Wang, Z., and de Freitas, N. ().
Playing hard exploration games by watching youtube.
In
Neural Information Processing Systems
.Babaeizadeh et al.
Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R. H., and Levine, S. ().
Stochastic variational video prediction..
Bagad et al.
Bagad, P., Tapaswi, M., and Snoek, C. G. M. ().
Test of time: Instilling video-language models with a sense of time..
Bahl et al.
Bahl, S., Gupta, A., and Pathak, D. ().
Human-to-robot imitation in the wild.
ArXiv
,abs/.
.Bahl et al.
Bahl, S., Mendonca, R., Chen, L., Jain, U., and Pathak, D. ().
Affordances from human videos as a versatile representation for robotics..
Bai et al.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. ().
Constitutional ai: Harmlessness from ai feedback..
Bain et al.
Bain, M., Nagrani, A., Varol, G., and Zisserman, A. ().
Frozen in time: A joint video and image encoder for end-to-end retrieval.
In
Proceedings of the IEEE/CVF International Conference on Computer Vision
, pp. .Baker et al.
Baker, B., Akkaya, I., Zhokhov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J. ().
Video pretraining (vpt): Learning to act by watching unlabeled online videos.
ArXiv
,abs/.
.Balažević et al.
Balažević, I., Shi, Y., Papalampidi, P., Chaabouni, R., Koppula, S., and Hénaff, O. J. ().
Memory consolidation enables long-context video understanding..
Bansal et al.
Bansal, H., Bitton, Y., Szpektor, I., Chang, K.-W., and Grover, A. ().
Videocon: Robust video-language alignment via contrast captions..
Bar-Tal et al.
Bar-Tal, O., Chefer, H., Tov, O., Herrmann, C., Paiss, R., Zada, S., Ephrat, A., Hur, J., Li, Y., Michaeli, T., et al. ().
Lumiere: A space-time diffusion model for video generation..
Bardes et al.
Bardes, A., Garrido, Q., Ponce, J., Chen, X., Rabbat, M., LeCun, Y., Assran, M., and Ballas, N. ().
V-jepa: Latent video prediction for visual representation learning..
Baumli et al.
Baumli, K., Baveja, S., Behbahani, F., Chan, H., Comanici, G., Flennerhag, S., Gazeau, M., Holsheimer, K., Horgan, D., Laskin, M., et al. ().
Vision-language models as a source of rewards..
Belkhale et al.
Belkhale, S., Ding, T., Xiao, T., Sermanet, P., Vuong, Q., Tompson, J., Chebotar, Y., Dwibedi, D., and Sadigh, D. ().
Rt-h: Action hierarchies using language..
Bellemare et al.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. ().
The arcade learning environment: An evaluation platform for general agents.
Journal of Artificial Intelligence Research
,47
, 253279.Berrevoets et al.
Berrevoets, J., Kacprzyk, K., Qian, Z., and van der Schaar, M. ().
Causal deep learning..
Bertasius et al.
Bertasius, G., Wang, H., and Torresani, L. ().
Is space-time attention all you need for video understanding?.
In
ICML
, Vol. 2, p. 4.Betker et al.
Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., et al. ().
Improving image generation with better captions.
Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf
,2
(3), 8.Bharadhwaj et al.
Bharadhwaj, H., Gupta, A., Kumar, V., and Tulsiani, S. ().
Towards generalizable zero-shot manipulation via translating human interaction plans.
ArXiv
,abs/.
.Bhat et al.
Bhat, S. F., Birkl, R., Wofk, D., Wonka, P., and Müller, M. ().
Zoedepth: Zero-shot transfer by combining relative and metric depth..
Bhateja et al.
Bhateja, C., Guo, D., Ghosh, D., Singh, A., Tomar, M., Vuong, Q. H., Chebotar, Y., Levine, S., and Kumar, A. ().
Robotic offline rl from internet videos via value-function pre-training.
ArXiv
,abs/.
.Black et al.
Black, K., Janner, M., Du, Y., Kostrikov, I., and Levine, S. (a).
Training diffusion models with reinforcement learning..
Black et al.
Black, K., Nakamoto, M., Atreya, P., Walke, H., Finn, C., Kumar, A., and Levine, S. (b).
Zero-shot robotic manipulation with pretrained image-editing diffusion models.
ArXiv
,abs/.
.Blank et al.
Blank, N., Reuss, M., Wenzel, F., Mees, O., and Lioutikov, R. ().
Scaling robot policy learning via zero-shot labeling with foundation models.
In
2nd Workshop on Mobile Manipulation and Embodied Intelligence at ICRA
.Blattmann et al.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. (a).
Stable video diffusion: Scaling latent video diffusion models to large datasets..
Blattmann et al.
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. (b).
Align your latents: High-resolution video synthesis with latent diffusion models.
In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
, pp. .Bousmalis et al.
Bousmalis, K., Vezzani, G., Rao, D., Devin, C., Lee, A. X., Bauza, M., Davchev, T., Zhou, Y., Gupta, A., Raju, A., et al. ().
Robocat: A self-improving foundation agent for robotic manipulation..
Brohan et al.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Choromanski, K., Ding, T., Driess, D., Finn, C., Florence, P. R., Fu, C., Arenas, M. G., Gopalakrishnan, K., Han, K., Hausman, K., Herzog, A., Hsu, J., Ichter, B., Irpan, A., Joshi, N. J., Julian, R. C., Kalashnikov, D., Kuang, Y., Leal, I., Levine, S., Michalewski, H., Mordatch, I., Pertsch, K., Rao, K., Reymann, K., Ryoo, M. S., Salazar, G., Sanketi, P. R., Sermanet, P., Singh, J., Singh, A., Soricut, R., Tran, H., Vanhoucke, V., Vuong, Q. H., Wahid, A., Welker, S., Wohlhart, P., Xiao, T., Yu, T., and Zitkovich, B. ().
Rt-2: Vision-language-action models transfer web knowledge to robotic control.
ArXiv
,abs/.
.Brohan et al.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jackson, T., Jesmonth, S., Joshi, N. J., Julian, R. C., Kalashnikov, D., Kuang, Y., Leal, I., Lee, K.-H., Levine, S., Lu, Y., Malla, U., Manjunath, D., Mordatch, I., Nachum, O., Parada, C., Peralta, J., Perez, E., Pertsch, K., Quiambao, J., Rao, K., Ryoo, M. S., Salazar, G., Sanketi, P. R., Sayed, K., Singh, J., Sontakke, S. A., Stone, A., Tan, C., Tran, H., Vanhoucke, V., Vega, S., Vuong, Q. H., Xia, F., Xiao, T., Xu, P., Xu, S., Yu, T., and Zitkovich, B. ().
Rt-1: Robotics transformer for real-world control at scale.
ArXiv
,abs/.
.Brooks et al.
Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., and Ramesh, A. ().
Video generation models as world simulators..
Brown et al.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T. J., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. ().
Language models are few-shot learners.
ArXiv
,abs/.
.Bruce et al.
Bruce, J., Dennis, M., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steigerwald, R., Apps, C., Aytar, Y., Bechtle, S., Behbahani, F. M. P., Chan, S., Heess, N. M. O., Gonzalez, L., Osindero, S., Ozair, S., Reed, S., Zhang, J., Zolna, K., Clune, J., de Freitas, N., Singh, S., and Rocktaschel, T. ().
Genie: Generative interactive environments..
Burda et al.
Burda, Y., Edwards, H., Storkey, A. J., and Klimov, O. ().
Exploration by random network distillation.
ArXiv
,abs/.
.Burns et al.
Burns, K., Witzel, Z., Hamid, J. I., Yu, T., Finn, C., and Hausman, K. ().
What makes pre-trained visual representations successful for robust manipulation?.
ArXiv
,abs/.
.Caesar et al.
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. ().
nuscenes: A multimodal dataset for autonomous driving..
Cai et al.
Cai, S., Zhang, B., Wang, Z., Ma, X., Liu, A., and Liang, Y. ().
Groot: Learning to follow instructions by watching gameplay videos.
ArXiv
,abs/.
.Campos et al.
Campos, V., Sprechmann, P., Hansen, S., Barreto, A., Kapturowski, S., Vitvitskyi, A., Badia, A. P., and Blundell, C. ().
Beyond fine-tuning: Transferring behavior in reinforcement learning..
Chane-Sane et al.
Chane-Sane, E., Schmid, C., and Laptev, I. ().
Learning video-conditioned policies for unseen manipulation tasks..
Chang et al.
Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. ().
Maskgit: Masked generative image transformer..
Chang et al.
Chang, M., Gupta, A., and Gupta, S. ().
Semantic visual navigation by watching youtube videos.
ArXiv
,abs/.
.Chang et al.
Chang, M., Gupta, A., and Gupta, S. ().
Learning value functions from undirected state-only experience.
ArXiv
,abs/.
.Chang et al.
Chang, M., Prakash, A., and Gupta, S. ().
Look ma, no hands! agent-environment factorization of egocentric videos.
ArXiv
,abs/.
.Chebotar et al.
Chebotar, Y., Vuong, Q., Hausman, K., Xia, F., Lu, Y., Irpan, A., Kumar, A., Yu, T., Herzog, A., Pertsch, K., et al. ().
Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions.
In
Conference on Robot Learning
, pp. . PMLR.Chen et al.
Chen, A. S., Nair, S., and Finn, C. ().
Learning generalizable robotic reward functions from in-the-wild human videos.
ArXiv
,abs/.
.Chen et al.
Chen, G., Zheng, Y.-D., Wang, J., Xu, J., Huang, Y., Pan, J., Wang, Y., Wang, Y., Qiao, Y., Lu, T., et al. (a).
Videollm: Modeling video sequence with large language models..
Chen et al.
Chen, J., Zhu, D., Haydarov, K., Li, X., and Elhoseiny, M. (b).
Video chatcaptioner: Towards the enriched spatiotemporal descriptions..
Chen et al.
Chen, W., Wu, J., Xie, P., Wu, H., Li, J., Xia, X., Xiao, X., and Lin, L. (c).
Control-a-video: Controllable text-to-video generation with diffusion models..
Chen et al.
Chen, X., Xie, S., and He, K. ().
An empirical study of training self-supervised vision transformers..
Chevalier-Boisvert et al.
Chevalier-Boisvert, M., Dai, B., Towers, M., Perez-Vicente, R., Willems, L., Lahlou, S., Pal, S., Castro, P. S., and Terry, J. ().
Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks.
Advances in Neural Information Processing Systems
,36
.Chiappa et al.
Chiappa, S., Racaniere, S., Wierstra, D., and Mohamed, S. ().
Recurrent environment simulators..
Clark et al.
Clark, A., Donahue, J., and Simonyan, K. ().
Adversarial video generation on complex datasets..
Cui et al.
Cui, Z. J., Wang, Y., Shafiullah, N. M. M., and Pinto, L. ().
From play to policy: Conditional behavior generation from uncurated robot data.
ArXiv
,abs/.
.Dai et al.
Dai, X., Hou, J., Ma, C.-Y., Tsai, S., Wang, J., Wang, R., Zhang, P., Vandenhende, S., Wang, X., Dubey, A., et al. ().
Emu: Enhancing image generation models using photogenic needles in a haystack..
Damen et al.
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. ().
Scaling egocentric vision: The epic-kitchens dataset.
In
Proceedings of the European conference on computer vision (ECCV)
, pp. 720736.Damen et al.
Damen, D., Doughty, H., Farinella, G. M., Furnari, A., Kazakos, E., Ma, J., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. ().
Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100..
Dao et al.
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. ().
Flashattention: Fast and memory-efficient exact attention with io-awareness.
Advances in Neural Information Processing Systems
,35
, .Darkhalil et al.
Darkhalil, A., Shan, D., Zhu, B., Ma, J., Kar, A., Higgins, R., Fidler, S., Fouhey, D., and Damen, D. ().
Epic-kitchens visor benchmark: Video segmentations and object relations.
Advances in Neural Information Processing Systems
,35
, .Dasari et al.
Dasari, S., Srirama, M. K., Jain, U., and Gupta, A. ().
An unbiased look at datasets for visuo-motor pre-training.
ArXiv
,abs/.
.Deitke et al.
Deitke, M., VanderBilt, E., Herrasti, A., Weihs, L., Ehsani, K., Salvador, J., Han, W., Kolve, E., Kembhavi, A., and Mottaghi, R. ().
Procthor: Large-scale embodied ai using procedural generation.
Advances in Neural Information Processing Systems
,35
, .Deng et al.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ().
Imagenet: A large-scale hierarchical image database.
In
IEEE conference on computer vision and pattern recognition
, pp. 248255. Ieee.Denton and Fergus
Denton, E., and Fergus, R. ().
Stochastic video generation with a learned prior.
In
International conference on machine learning
, pp. . PMLR.Ding et al.
Ding, H., Liu, C., He, S., Jiang, X., Torr, P. H., and Bai, S. (a).
Mose: A new dataset for video object segmentation in complex scenes.
In
Proceedings of the IEEE/CVF International Conference on Computer Vision
, pp. .Ding et al.
Ding, Z., Luo, H., Li, K., Yue, J., Huang, T., and Lu, Z. (b).
Clip4mc: An rl-friendly vision-language model for minecraft.
ArXiv
,abs/.
.Du and Kaelbling
Du, Y., and Kaelbling, L. P. ().
Compositional generative modeling: A single model is not all you need.
ArXiv
,abs/.
.Du et al.
Du, Y., Yang, M., Dai, B., Dai, H., Nachum, O., Tenenbaum, J. B., Schuurmans, D., and Abbeel, P. (a).
Learning universal policies via text-guided video generation.
ArXiv
,abs/.
.Du et al.
Du, Y., Yang, M., Florence, P. R., Xia, F., Wahid, A., Ichter, B., Sermanet, P., Yu, T., Abbeel, P., Tenenbaum, J., Kaelbling, L. P., Zeng, A., and Tompson, J. (b).
Video language planning.
ArXiv
,abs/.
.Du et al.
Du, Y., Konyushkova, K., Denil, M., Raju, A., Landon, J., Hill, F., de Freitas, N., and Cabi, S. (c).
Vision-language models as success detectors..
Edwards and Isbell
Edwards, A. D., and Isbell, C. L. ().
Perceptual values from observation..
Edwards et al.
Edwards, A. D., Sahni, H., Liu, R., Hung, J., Jain, A., Wang, R., Ecoffet, A., Miconi, T., Isbell, C. L., and Yosinski, J. ().
Estimating q(s, s) with deep deterministic dynamics gradients.
In
International Conference on Machine Learning
.Edwards et al.
Edwards, A. D., Sahni, H., Schroecker, Y., and Isbell, C. L. ().
Imitating latent policies from observation.
ArXiv
,abs/.
.Emmons et al.
Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, S. ().
Rvs: What is essential for offline rl via supervised learning?..
Escontrela et al.
Escontrela, A., Adeniji, A., Yan, W., Jain, A., Peng, X. B., Goldberg, K., Lee, Y., Hafner, D., and Abbeel, P. ().
Video prediction models as rewards for reinforcement learning.
ArXiv
,abs/.
.Esser et al.
Esser, P., Rombach, R., and Ommer, B. ().
Taming transformers for high-resolution image synthesis..
Eze and Crick
Eze, C., and Crick, C. ().
Learning by watching: A review of video-based learning approaches for robot manipulation..
Faldor et al.
Faldor, M., Zhang, J., Cully, A., and Clune, J. ().
Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code..
Fan et al.
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. ().
Minedojo: Building open-ended embodied agents with internet-scale knowledge.
Advances in Neural Information Processing Systems
,35
, .Finn et al.
Finn, C., Abbeel, P., and Levine, S. ().
Model-agnostic meta-learning for fast adaptation of deep networks.
In
International conference on machine learning
, pp. . PMLR.Gao et al.
Gao, Z., Tan, C., Wu, L., and Li, S. Z. ().
Simvp: Simpler yet better video prediction.
In
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
, pp. .Garcia et al.
Garcia, C. E., Prett, D. M., and Morari, M. ().
Model predictive control: Theory and practicea survey.
Automatica
,25
(3), 335348.Gavenski et al.
Gavenski, N., Rodrigues, O., and Luck, M. ().
Imitation learning: A survey of learning methods, environments and metrics..
Ge et al.
Ge, S., Hayes, T., Yang, H., Yin, X., Pang, G., Jacobs, D., Huang, J.-B., and Parikh, D. ().
Long video generation with time-agnostic vqgan and time-sensitive transformer.
In
European Conference on Computer Vision
, pp. 102118. Springer.Ge et al.
Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Jacobs, D., Huang, J.-B., Liu, M.-Y., and Balaji, Y. ().
Preserve your own correlation: A noise prior for video diffusion models.
In
Proceedings of the IEEE/CVF International Conference on Computer Vision
, pp. .Ghosh et al.
Ghosh, D., Bhateja, C. A., and Levine, S. ().
Reinforcement learning from passive data via latent intentions.
In
International Conference on Machine Learning
, pp. . PMLR.Girdhar et al.
Girdhar, R., El-Nouby, A., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. ().
Omnimae: Single model masked pretraining on images and videos..
Girdhar et al.
Girdhar, R., Ramanan, D., Gupta, A., Sivic, J., and Russell, B. ().
Actionvlad: Learning spatio-temporal aggregation for action classification.
In
Proceedings of the IEEE conference on computer vision and pattern recognition
, pp. 971980.Goyal et al.
Goyal, R., Kahou, S. E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fründ, I., Yianilos, P. N., Mueller-Freitag, M., Hoppe, F., Thurau, C., Bax, I., and Memisevic, R. ().
The something something video database for learning and evaluating visual common sense..
Grauman et al.
Grauman, K., Westbury, A., Byrne, E., Chavis, Z. Q., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., Martin, M., Nagarajan, T., Radosavovic, I., Ramakrishnan, S. K., Ryan, F., and et. al., J. S. ().
Ego4d: Around the world in 3,000 hours of egocentric video..
Grauman et al.
Grauman, K., Westbury, A., Torresani, L., Kitani, K., Malik, J., Afouras, T., Ashutosh, K., Baiyya, V., Bansal, S., Boote, B., et al. ().
Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives..
Grill et al.
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. ().
Bootstrap your own latent-a new approach to self-supervised learning.
Advances in neural information processing systems
,33
, .Gu and Dao
Gu, A., and Dao, T. ().
Mamba: Linear-time sequence modeling with selective state spaces..
Gu et al.
Gu, J., Xiang, F., Li, X., Ling, Z., Liu, X., Mu, T., Tang, Y., Tao, S., Wei, X., Yao, Y., et al. ().
Maniskill2: A unified benchmark for generalizable manipulation skills..
Guan et al.
Guan, L., Zhou, Y., Liu, D., Zha, Y., Amor, H. B., and Kambhampati, S. ().
task success is not enough: Investigating the use of video-language models as behavior critics for catching undesirable agent behaviors..
Gupta et al.
Gupta, A., Tian, S., Zhang, Y., Wu, J., Martín-Martín, R., and Fei-Fei, L. ().
Maskvit: Masked visual pre-training for video prediction..
Ha et al.
Ha, H., Florence, P., and Song, S. ().
Scaling up and distilling down: Language-guided robot skill acquisition.
In
Conference on Robot Learning
, pp. . PMLR.Haarnoja et al.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. ().
Soft actor-critic algorithms and applications..
Hafner et al.
Hafner, D., Lee, K.-H., Fischer, I., and Abbeel, P. ().
Deep hierarchical planning from pixels.
Advances in Neural Information Processing Systems
,35
, .Hafner et al.
Hafner, D., Lillicrap, T. P., Fischer, I. S., Villegas, R., Ha, D. R., Lee, H., and Davidson, J. ().
Learning latent dynamics for planning from pixels.
ArXiv
,abs/.
.Hafner et al.
Hafner, D., Paukonis, J., Ba, J., and Lillicrap, T. P. ().
Mastering diverse domains through world models.
ArXiv
,abs/.
.Hawthorne et al.
Hawthorne, C., Jaegle, A., Cangea, C., Borgeaud, S., Nash, C., Malinowski, M., Dieleman, S., Vinyals, O., Botvinick, M., Simon, I., et al. ().
General-purpose, long-context autoregressive modeling with perceiver ar.
In
International Conference on Machine Learning
, pp. . PMLR.He et al.
He, H., Bai, C., Pan, L., Zhang, W., Zhao, B., and Li, X. ().
Large-scale actionless video pre-training via discrete diffusion for efficient policy learning..
He et al.
He, K., Zhang, X., Ren, S., and Sun, J. ().
Deep residual learning for image recognition.
In
Proceedings of the IEEE conference on computer vision and pattern recognition
, pp. 770778.Ho et al.
Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. ().
Imagen video: High definition video generation with diffusion models..
Ho and Ermon
Ho, J., and Ermon, S. ().
Generative adversarial imitation learning.
Advances in neural information processing systems
,29
.Ho et al.
Ho, J., Jain, A., and Abbeel, P. ().
Denoising diffusion probabilistic models.
ArXiv
,abs/.
.Ho et al.
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. ().
Video diffusion models.
Advances in Neural Information Processing Systems
,35
, .Höppe et al.
Höppe, T., Mehrjou, A., Bauer, S., Nielsen, D., and Dittadi, A. ().
Diffusion models for video prediction and infilling..
Hu et al.
Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., and Corrado, G. ().
Gaia-1: A generative world model for autonomous driving.
ArXiv
,abs/.
.Hu et al.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. ().
Lora: Low-rank adaptation of large language models..
Hu et al.
Hu, Y., Xie, Q., Jain, V., Francis, J., Patrikar, J., Keetha, N., Kim, S., Xie, Y., Zhang, T., Zhao, Z., et al. (a).
Toward general-purpose robots via foundation models: A survey and meta-analysis..
Hu et al.
Hu, Y., Wang, R., Li, L., and Gao, Y. (b).
For pre-trained vision models in motor control, not all policy learning methods are created equal.
In
International Conference on Machine Learning
.Huang et al.
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. (a).
Large language models cannot self-correct reasoning yet..
Huang et al.
Huang, T., Jiang, G., Ze, Y., and Xu, H. (b).
Diffusion reward: Learning rewards via conditional video diffusion..
Huang et al.
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P. R., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., Sermanet, P., Brown, N., Jackson, T., Luu, L., Levine, S., Hausman, K., and Ichter, B. ().
Inner monologue: Embodied reasoning through planning with language models.
In
Conference on Robot Learning
.Ibarz et al.
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., and Levine, S. ().
How to train your robot with deep reinforcement learning: lessons we have learned.
The International Journal of Robotics Research
,40
(4-5), 698721.Jain et al.
Jain, V., Attarian, M., Joshi, N. J., Wahid, A., Driess, D., Vuong, Q., Sanketi, P. R., Sermanet, P., Welker, S., Chan, C., et al. ().
Vid2robot: End-to-end video-conditioned policy learning with cross-attention transformers..
Jang
Jang, E. ().
All neural networks, all autonomous, all 1x speed.
https://www.1x.tech/discover/all-neural-networks-all-autonomous-all-1x-speed.Accessed: -04-10.
Janner et al.
Janner, M., Fu, J., Zhang, M., and Levine, S. ().
When to trust your model: Model-based policy optimization.
ArXiv
,abs/.
.Jiang et al.
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. ().
Mixtral of experts..
Jiang et al.
Jiang, J., Deng, F., Singh, G., and Ahn, S. ().
Object-centric slot diffusion..
Jin et al.
Jin, Y., Xu, K., Chen, L., Liao, C., Tan, J., Chen, B., Lei, C., Liu, A., Song, C., Lei, X., et al. ().
Unified language-vision pretraining with dynamic discrete visual tokenization..
Jing et al.
Jing, Y., Zhu, X., Liu, X., Sima, Q., Yang, T., Feng, Y., and Kong, T. ().
Exploring visual pre-training for robot manipulation: Datasets, models and methods.
ArXiv
,abs/.
.JyothirS et al.
JyothirS, V., Jalagam, S., LeCun, Y., and Sobal, V. ().
Gradient-based planning with world models.
ArXiv
,abs/.
.Kaplan et al.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. ().
Scaling laws for neural language models..
Karaev et al.
Karaev, N., Rocco, I., Graham, B., Neverova, N., Vedaldi, A., and Rupprecht, C. ().
Cotracker: It is better to track together..
Karamcheti et al.
Karamcheti, S., Nair, S., Chen, A. S., Kollar, T., Finn, C., Sadigh, D., and Liang, P. ().
Language-driven representation learning for robotics.
ArXiv
,abs/.
.Karnan et al.
Karnan, H., Warnell, G., Xiao, X., and Stone, P. ().
Voila: Visual-observation-only imitation learning for autonomous navigation..
Kaufmann et al.
Kaufmann, E., Bauersfeld, L., Loquercio, A., Müller, M., Koltun, V., and Scaramuzza, D. (a).
Champion-level drone racing using deep reinforcement learning.
Nature
,620
(), 982987.Kaufmann et al.
Kaufmann, T., Weng, P., Bengs, V., and Hüllermeier, E. (b).
A survey of reinforcement learning from human feedback..
Kay et al.
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. ().
The kinetics human action video dataset..
Khazatsky et al.
Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., Karamcheti, S., Nasiriany, S., Srirama, M. K., Chen, L. Y., Ellis, K., Fagan, P., Hejna, J., Itkina, M., Lepert, M., Ma, Y., Miller, P. T., Wu, J., Belkhale, S., Dass, S., Ha, H., Jain, A., Lee, A., Lee, Y., Memmel, M., Park, S. Y., and et. al., I. R. ().
Droid: A large-scale in-the-wild robot manipulation dataset.
ArXiv
,abs/.
.Kim et al.
Kim, M. J., Wu, J., and Finn, C. ().
Giving robots a hand: Learning generalizable manipulation with eye-in-hand human video demonstrations.
ArXiv
,abs/.
.Kirillov et al.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. ().
Segment anything.
In
Proceedings of the IEEE/CVF International Conference on Computer Vision
, pp. .Kirk et al.
Kirk, R., Zhang, A., Grefenstette, E., and Rocktäschel, T. ().
A survey of zero-shot generalisation in deep reinforcement learning.
Journal of Artificial Intelligence Research
,76
, 201264.Klissarov et al.
Klissarov, M., DOro, P., Sodhani, S., Raileanu, R., Bacon, P.-L., Vincent, P., Zhang, A., and Henaff, M. ().
Motif: Intrinsic motivation from artificial intelligence feedback..
Ko et al.
Ko, P.-C., Mao, J., Du, Y., Sun, S.-H., and Tenenbaum, J. ().
Learning to act from actionless videos through dense correspondences.
ArXiv
,abs/.
.Kober et al.
Kober, J., Bagnell, J. A., and Peters, J. ().
Reinforcement learning in robotics: A survey.
The International Journal of Robotics Research
,32
(11), .Kolve et al.
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. ().
Ai2-thor: An interactive 3d environment for visual ai..
Kondratyuk et al.
Kondratyuk, D., Yu, L., Gu, X., Lezama, J., Huang, J., Hornung, R., Adam, H., Akbari, H., Alon, Y., Birodkar, V., et al. ().
Videopoet: A large language model for zero-shot video generation..
Krizhevsky et al.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. ().
Imagenet classification with deep convolutional neural networks.
Advances in neural information processing systems
,25
.Kroemer et al.
Kroemer, O., Niekum, S., and Konidaris, G. ().
A review of robot learning for manipulation: Challenges, representations, and algorithms.
Journal of machine learning research
,22
(30), 182.Krotkov et al.
Krotkov, E., Hackett, D., Jackel, L., Perschbacher, M., Pippine, J., Strauss, J., Pratt, G., and Orlowski, C. ().
The darpa robotics challenge finals: Results and perspectives..
Kumar et al.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. ().
Conservative q-learning for offline reinforcement learning.
Advances in Neural Information Processing Systems
,33
, .Kumar et al.
Kumar, S., Zamora, J., Hansen, N., Jangir, R., and Wang, X. ().
Graph inverse reinforcement learning from diverse videos.
In
Conference on Robot Learning
.Kumar et al.
Kumar, V., Shah, R., Zhou, G., Moens, V., Caggiano, V., Gupta, A., and Rajeswaran, A. ().
Robohive: A unified framework for robot learning.
Advances in Neural Information Processing Systems
,36
.Küttler et al.
Küttler, H., Nardelli, N., Miller, A., Raileanu, R., Selvatici, M., Grefenstette, E., and Rocktäschel, T. ().
The nethack learning environment.
Advances in Neural Information Processing Systems
,33
, .Lanham et al.
Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., et al. ().
Measuring faithfulness in chain-of-thought reasoning..
Laskin et al.
Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S. S., Filos, A., Brooks, E. A., Gazeau, M., Sahni, H., Singh, S., and Mnih, V. ().
In-context reinforcement learning with algorithm distillation.
ArXiv
,abs/.
.LeCun
LeCun, Y. ().
A path towards autonomous machine intelligence version 0.9. 2, -06-27.
Open Review
,62
(1).Lee et al.
Lee, J., Xie, A., Pacchiano, A., Chandak, Y., Finn, C., Nachum, O., and Brunskill, E. ().
Supervised pretraining can learn in-context reinforcement learning.
ArXiv
,abs/.
.Levine et al.
Levine, S., Kumar, A., Tucker, G., and Fu, J. ().
Offline reinforcement learning: Tutorial, review, and perspectives on open problems.
ArXiv
,abs/.
.Li et al.
Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Yang, J., Li, C., and Liu, Z. ().
Mimic-it: Multi-modal in-context instruction tuning..
Li et al.
Li, C., Zhang, R., Wong, J., Gokmen, C., Srivastava, S., Martín-Martín, R., Wang, C., Levine, G., Ai, W., Martinez, B., et al. ().
Behavior-1k: A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation..
Li et al.
Li, J., Li, D., Xiong, C., and Hoi, S. ().
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.
In
International conference on machine learning
, pp. . PMLR.Li et al.
Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., and Qiao, Y. ().
Videochat: Chat-centric video understanding..
Li et al.
Li, K., Li, X., Wang, Y., He, Y., Wang, Y., Wang, L., and Qiao, Y. ().
Videomamba: State space model for efficient video understanding..
Li et al.
Li, K., Wang, Y., Li, Y., Wang, Y., He, Y., Wang, L., and Qiao, Y. ().
Unmasked teacher: Towards training-efficient video foundation models..
Li et al.
Li, P., Liu, T., Li, Y., Han, M., Geng, H., Wang, S., Zhu, Y., Zhu, S.-C., and Huang, S. ().
Ag2manip: Learning novel manipulation skills with agent-agnostic visual and action representations..
Li et al.
Li, S., Du, Y., Tenenbaum, J. B., Torralba, A., and Mordatch, I. ().
Composing ensembles of pre-trained models via iterative consensus..
Liang et al.
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. ().
Code as policies: Language model programs for embodied control.
In
IEEE International Conference on Robotics and Automation (ICRA)
, pp. . IEEE.Lifshitz et al.
Lifshitz, S., Paster, K., Chan, H., Ba, J., and McIlraith, S. A. ().
Steve-1: A generative model for text-to-behavior in minecraft.
ArXiv
,abs/.
.Lillicrap et al.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. ().
Continuous control with deep reinforcement learning..
Lin et al.
Lin, B., Zhu, B., Ye, Y., Ning, M., Jin, P., and Yuan, L. ().
Video-llava: Learning united visual representation by alignment before projection..
Lin et al.
Lin, H., Peng, S., Xu, Z., Yan, Y., Shuai, Q., Bao, H., and Zhou, X. ().
Efficient neural radiance fields for interactive free-viewpoint video.
In
SIGGRAPH Asia Conference Papers
, pp. 19.Liu et al.
Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q., Zhu, Y., and Stone, P. (a).
Libero: Benchmarking knowledge transfer for lifelong robot learning.
Advances in Neural Information Processing Systems
,36
.Liu et al.
Liu, H., Yan, W., Zaharia, M., and Abbeel, P. (b).
World model on million-length video and language with blockwise ringattention..
Liu et al.
Liu, H., Zaharia, M., and Abbeel, P. (a).
Ring attention with blockwise transformers for near-infinite context..
Liu et al.
Liu, H., Zhuge, M., Li, B., Wang, Y., Faccio, F., Ghanem, B., and Schmidhuber, J. (b).
Learning to identify critical states for reinforcement learning from videos.
In
Proceedings of the IEEE/CVF International Conference on Computer Vision
, pp. .Liu et al.
Liu, J., Yuan, Y., Hao, J., Ni, F., Fu, L., Chen, Y., and Zheng, Y. ().
Enhancing robotic manipulation with ai feedback from multimodal large language models..
Liu et al.
Liu, M., Zhu, Z., Zhuang, Y., Zhang, W., Hao, J., Yu, Y., and Wang, J. ().
Plan your target and learn your skills: Transferable state-only imitation learning via decoupled policy optimization.
In
International Conference on Machine Learning
.Loper et al.
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., and Black, M. J. ().
Smpl: A skinned multi-person linear model.
In
Seminal Graphics Papers: Pushing the Boundaries, Volume 2
, pp. 851866.Lotter et al.
Lotter, W., Kreiman, G., and Cox, D. ().
Deep predictive coding networks for video prediction and unsupervised learning..
Lynch et al.
Lynch, C., Khansari, M., Xiao, T., Kumar, V., Tompson, J., Levine, S., and Sermanet, P. ().
Learning latent plans from play.
Fuxin Intelligent Product Page
In
Conference on robot learning
, pp. . PMLR.Lynch et al.
Lynch, C., Wahid, A., Tompson, J., Ding, T., Betker, J., Baruch, R., Armstrong, T., and Florence, P. ().
Interactive language: Talking to robots in real time..
Ma et al.
Ma, Y. J., Liang, W. J., Som, V., Kumar, V., Zhang, A., Bastani, O., and Jayaraman, D. ().
Liv: Language-image representations and rewards for robotic control.
In
International Conference on Machine Learning
.Ma et al.
Ma, Y. J., Sodhani, S., Jayaraman, D., Bastani, O., Kumar, V., and Zhang, A. ().
Vip: Towards universal visual reward and representation via value-implicit pre-training.
ArXiv
,abs/.
.Maaz et al.
Maaz, M., Rasheed, H., Khan, S., and Khan, F. S. ().
Video-chatgpt: Towards detailed video understanding via large vision and language models..
Mack
Mack, C. A. ().
Fifty years of moores law.
IEEE Transactions on semiconductor manufacturing
,24
(2), 202207.Madan et al.
Madan, N., Moegelmose, A., Modi, R., Rawat, Y. S., and Moeslund, T. B. ().
Foundation models for video understanding: A survey..
Majumdar et al.
Majumdar, A., Yadav, K., Arnaud, S., Ma, Y. J., Chen, C., Silwal, S., Jain, A., Berges, V.-P., Abbeel, P., Malik, J., Batra, D., Lin, Y., Maksymets, O., Rajeswaran, A., and Meier, F. ().
Where are we in the search for an artificial visual cortex for embodied intelligence?.
ArXiv
,abs/.
.Makoviychuk et al.
Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., et al. ().
Isaac gym: High performance gpu-based physics simulation for robot learning..
Mandikal and Grauman
Mandikal, P., and Grauman, K. ().
Dexvip: Learning dexterous grasping with human hand pose priors from video.
ArXiv
,abs/.
.Mees et al.
Mees, O., Hermann, L., Rosete-Beas, E., and Burgard, W. ().
Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks.
IEEE Robotics and Automation Letters
,7
(3), .Mendonca et al.
Mendonca, R., Bahl, S., and Pathak, D. ().
Structured world models from human videos.
ArXiv
,abs/.
.Miech et al.
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., and Sivic, J. ().
Howto100m: Learning a text-video embedding by watching hundred million narrated video clips..
Ming et al.
Ming, R., Huang, Z., Ju, Z., Hu, J., Peng, L., and Zhou, S. ().
A survey on video prediction: From deterministic to generative approaches.
ArXiv
,abs/.
.Moore
Moore, G. E. ().
Cramming more components onto integrated circuits.
Proceedings of the IEEE
,86
(1), 8285.Mu et al.
Mu, Y., Zhang, Q., Hu, M., Wang, W., Ding, M., Jin, J., Wang, B., Dai, J., Qiao, Y., and Luo, P. ().
Embodiedgpt: Vision-language pre-training via embodied chain of thought.
ArXiv
,abs/.
.Nagarajan and Grauman
Nagarajan, T., and Grauman, K. ().
Shaping embodied agent behavior with activity-context priors from egocentric video.
ArXiv
,abs/.
.Nagrani et al.
Nagrani, A., Seo, P. H., Seybold, B., Hauth, A., Manen, S., Sun, C., and Schmid, C. ().
Learning audio-video modalities from image captions.
In
European Conference on Computer Vision
, pp. 407426. Springer.Nair et al.
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A. ().
R3m: A universal visual representation for robot manipulation.
In
Conference on Robot Learning
.Narang and Chowdhery
Narang, S., and Chowdhery, A. ().
Pathways language model (palm): Scaling to 540 billion parameters for breakthrough performance..
Nasiriany et al.
Nasiriany, S., Xia, F., Yu, W., Xiao, T., Liang, J., Dasgupta, I., Xie, A., Driess, D., Wahid, A., Xu, Z., Vuong, Q. H., Zhang, T., Lee, T.-W. E., Lee, K.-H., Xu, P., Kirmani, S., Zhu, Y., Zeng, A., Hausman, K., Heess, N. M. O., Finn, C., Levine, S., and Ichter, B. ().
Pivot: Iterative visual prompting elicits actionable knowledge for vlms.
ArXiv
,abs/.
.Nauman et al.
Nauman, M., Bortkiewicz, M., Ostaszewski, M., Milos, P., Trzcinski, T., and Cygan, M. ().
Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning.
ArXiv
,abs/.
.Oord et al.
Oord, A. v. d., Li, Y., and Vinyals, O. ().
Representation learning with contrastive predictive coding..
OpenAI
OpenAI ().
Faulty reward functions.
https://openai.com/index/faulty-reward-functions/.Accessed: -05-23.
OpenAI
OpenAI ().
Gpt-4 technical report.
ArXiv
,abs/.
.Ouyang et al.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. ().
Training language models to follow instructions with human feedback.
Advances in neural information processing systems
,35
, .Padalkar et al.
Padalkar, A., Pooley, A., Jain, A., Bewley, A., Herzog, A., Irpan, A., Khazatsky, A., Rai, A., Singh, A., Brohan, A., et al. ().
Open x-embodiment: Robotic learning datasets and rt-x models..
Papalampidi et al.
Papalampidi, P., Koppula, S., Pathak, S., Chiu, J., Heyward, J., Patraucean, V., Shen, J., Miech, A., Zisserman, A., and Nematzdeh, A. ().
A simple recipe for contrastively pre-training video-first encoders beyond 16 frames.
ArXiv
,abs/.
.Parisi et al.
Parisi, S., Rajeswaran, A., Purushwalkam, S., and Gupta, A. K. ().
The unsurprising effectiveness of pre-trained vision models for control.
In
International Conference on Machine Learning
.Park et al.
Park, J. S., OBrien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. ().
Generative agents: Interactive simulacra of human behavior.
In
Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology
, pp. 122.Park et al.
Park, S., Ghosh, D., Eysenbach, B., and Levine, S. ().
Hiql: Offline goal-conditioned rl with latent states as actions.
Advances in Neural Information Processing Systems
,36
.Pathak et al.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. ().
Curiosity-driven exploration by self-supervised prediction..
Pavlakos et al.
Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A. A., Tzionas, D., and Black, M. J. ().
Expressive body capture: 3d hands, face, and body from a single image.
In
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
, pp. .Peng et al.
Peng, X. B., Coumans, E., Zhang, T., Lee, T.-W., Tan, J., and Levine, S. ().
Learning agile robotic locomotion skills by imitating animals..
Peng et al.
Peng, X. B., Kanazawa, A., Malik, J., Abbeel, P., and Levine, S. ().
Sfv: Reinforcement learning of physical skills from videos.
ACM Trans. Graph.
,37
, 178.Peri et al.
Peri, S., Lee, I., Kim, C., Fuxin, L., Hermans, T., and Lee, S. ().
Point cloud models improve visual robustness in robotic learners..
Pertsch et al.
Pertsch, K., Desai, R., Kumar, V., Meier, F., Lim, J. J., Batra, D., and Rai, A. ().
Cross-domain transfer via semantic skill imitation.
In
Conference on Robot Learning
.Pertsch et al.
Pertsch, K., Rybkin, O., Yang, J., Zhou, S., Derpanis, K. G., Daniilidis, K., Lim, J. J., and Jaegle, A. ().
Keyframing the future: Keyframe discovery for visual prediction and planning.
In
Conference on Learning for Dynamics & Control
.Peters et al.
Peters, J., Lee, D. D., Kober, J., Nguyen-Tuong, D., Bagnell, J. A., and Schaal, S. ().
Robot learning..
Popov et al.
Popov, I., Heess, N., Lillicrap, T., Hafner, R., Barth-Maron, G., Vecerik, M., Lampe, T., Tassa, Y., Erez, T., and Riedmiller, M. ().
Data-efficient deep reinforcement learning for dexterous manipulation..
Prudencio et al.
Prudencio, R. F., Maximo, M. R., and Colombini, E. L. ().
A survey on offline reinforcement learning: Taxonomy, review, and open problems..
Puig et al.
Puig, X., Undersander, E., Szot, A., Cote, M. D., Yang, T.-Y., Partsey, R., Desai, R., Clegg, A. W., Hlavac, M., Min, S. Y., et al. ().
Habitat 3.0: A co-habitat for humans, avatars and robots..
Pumacay et al.
Pumacay, W., Singh, I., Duan, J., Krishna, R., Thomason, J., and Fox, D. ().
The colosseum: A benchmark for evaluating generalization for robotic manipulation..
Qin et al.
Qin, Y., Su, H., and Wang, X. ().
From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation.
IEEE Robotics and Automation Letters
,7
, .Qin et al.
Qin, Y., Wu, Y.-H., Liu, S., Jiang, H., Yang, R., Fu, Y., and Wang, X. ().
Dexmv: Imitation learning for dexterous manipulation from human videos.
In
European Conference on Computer Vision
.Radford et al.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. ().
Learning transferable visual models from natural language supervision.
In
International conference on machine learning
, pp. . PMLR.Radford et al.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. ().
Language models are unsupervised multitask learners.
OpenAI blog
,1
(8), 9.Radosavovic et al.
Radosavovic, I., Xiao, T., James, S., Abbeel, P., Malik, J., and Darrell, T. ().
Real-world robot learning with masked visual pre-training.
In
Conference on Robot Learning
.Radosavovic et al.
Radosavovic, I., Zhang, B., Shi, B., Rajasegaran, J., Kamat, S., Darrell, T., Sreenath, K., and Malik, J. ().
Humanoid locomotion as next token prediction..
Ranasinghe et al.
Ranasinghe, K., Li, X., Kahatapitiya, K., and Ryoo, M. S. ().
Understanding long videos in one multimodal language model pass..
Ravichandar et al.
Ravichandar, H., Polydoros, A. S., Chernova, S., and Billard, A. ().
Recent advances in robot learning from demonstration.
Annu. Rev. Control. Robotics Auton. Syst.
,3
, 297330.Reed et al.
Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., Eccles, T., Bruce, J., Razavi, A., Edwards, A. D., Heess, N. M. O., Chen, Y., Hadsell, R., Vinyals, O., Bordbar, M., and de Freitas, N. ().
A generalist agent.
Trans. Mach. Learn. Res.
, .Romero et al.
Romero, J., Tzionas, D., and Black, M. J. ().
Embodied hands: Modeling and capturing hands and bodies together..
Rong et al.
Rong, Y., Shiratori, T., and Joo, H. ().
Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration..
Rosete-Beas et al.
Rosete-Beas, E., Mees, O., Kalweit, G., Boedecker, J., and Burgard, W. ().
Latent plans for task-agnostic offline reinforcement learning.
In
Conference on Robot Learning
.Rybkin et al.
Rybkin, O., Pertsch, K., Derpanis, K. G., Daniilidis, K., and Jaegle, A. ().
Learning what you can do before doing anything.
In
International Conference on Learning Representations
.Saxena et al.
Saxena, V., Ba, J., and Hafner, D. ().
Clockwork variational autoencoders.
Advances in Neural Information Processing Systems
,34
, .Schiappa et al.
Schiappa, M. C., Rawat, Y. S., and Shah, M. ().
Self-supervised learning for videos: A survey.
ACM Computing Surveys
,55
(13s), 137.Schmeckpeper et al.
Schmeckpeper, K., Rybkin, O., Daniilidis, K., Levine, S., and Finn, C. ().
Reinforcement learning with videos: Combining offline observations with interaction.
ArXiv
,abs/.
.Schmeckpeper et al.
Schmeckpeper, K., Xie, A., Rybkin, O., Tian, S., Daniilidis, K., Levine, S., and Finn, C. ().
Learning predictive models from observation and interaction.
In
European Conference on Computer Vision
.Schmidt and Jiang
Schmidt, D., and Jiang, M. ().
Learning to act without actions.
ArXiv
,abs/.
.Schonberger and Frahm
Schonberger, J. L., and Frahm, J.-M. ().
Structure-from-motion revisited.
In
Proceedings of the IEEE conference on computer vision and pattern recognition
, pp. .Schrittwieser et al.
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T. P., and Silver, D. ().
Mastering atari, go, chess and shogi by planning with a learned model.
Nature
,588
, 604 609.Schuhmann et al.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. ().
Laion-5b: An open large-scale dataset for training next generation image-text models.
Advances in Neural Information Processing Systems
,35
, .Schulman et al.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. ().
Proximal policy optimization algorithms..
Seo et al.
Seo, Y., Lee, K., James, S., and Abbeel, P. (a).
Reinforcement learning with action-free pre-training from videos.
ArXiv
,abs/.
.Seo et al.
Seo, Y., Lee, K., Liu, F., James, S., and Abbeel, P. (b).
Harp: Autoregressive latent video prediction with high-fidelity image generator..
Sermanet et al.
Sermanet, P., Ding, T., Zhao, J., Xia, F., Dwibedi, D., Gopalakrishnan, K., Chan, C., Dulac-Arnold, G., Maddineni, S., Joshi, N. J., et al. ().
Robovqa: Multimodal long-horizon reasoning for robotics..
Sermanet et al.
Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Schaal, S., Levine, S., and Brain, G. ().
Time-contrastive networks: Self-supervised learning from video.
In
IEEE international conference on robotics and automation (ICRA)
, pp. . IEEE.Shafiullah et al.
Shafiullah, N. M. M., Rai, A., Etukuru, H., Liu, Y., Misra, I., Chintala, S., and Pinto, L. ().
On bringing robots home..
Shah et al.
Shah, D., Osiński, B., Levine, S., et al. ().
Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action.
In
Conference on robot learning
, pp. 492504. PMLR.Shan et al.
Shan, D., Geng, J., Shu, M., and Fouhey, D. F. ().
Understanding human hands in contact at internet scale..
Shang et al.
Shang, C., You, A., Subramanian, S., Darrell, T., and Herzig, R. ().
Traveler: A multi-lmm agent framework for video question-answering..
Shang and Ryoo
Shang, J., and Ryoo, M. S. ().
Self-supervised disentangled representation learning for third-person imitation learning..
Shao et al.
Shao, L., Migimatsu, T., Zhang, Q., Yang, K., and Bohg, J. ().
Concept2robot: Learning manipulation concepts from instructions and human demonstrations.
The International Journal of Robotics Research
,40
, .Shaw et al.
Shaw, K., Bahl, S., and Pathak, D. ().
Videodex: Learning dexterity from internet videos.
In
Conference on Robot Learning
.Shi et al.
Shi, L. X., Hu, Z., Zhao, T. Z., Sharma, A., Pertsch, K., Luo, J., Levine, S., and Finn, C. ().
Yell at your robot: Improving on-the-fly from language corrections..
Sieb et al.
Sieb, M., Xian, Z., Huang, A., Kroemer, O., and Fragkiadaki, K. ().
Graph-structured visual imitation.
In
Conference on Robot Learning
, pp. 979989. PMLR.Silwal et al.
Silwal, S., Yadav, K., Wu, T., Vakil, J., Majumdar, A., Arnaud, S., Chen, C., Berges, V.-P., Batra, D., Rajeswaran, A., Kalakrishnan, M., Meier, F., and Maksymets, O. ().
What do we learn from a large-scale study of pre-trained visual representations in sim and real environments?.
ArXiv
,abs/.
.Simonyan and Zisserman
Simonyan, K., and Zisserman, A. ().
Two-stream convolutional networks for action recognition in videos.
Advances in neural information processing systems
,27
.Singer et al.
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al. ().
Make-a-video: Text-to-video generation without text-video data..
Sivakumar et al.
Sivakumar, A., Shaw, K., and Pathak, D. ().
Robotic telekinesis: Learning a robotic hand imitator by watching humans on youtube.
ArXiv
,abs/.
.Sjöberg
Sjöberg, A. ().
How many videos are there on youtube?.
https://www.dexerto.com/entertainment/how-many-videos-are-there-on-youtube-/.Smith et al.
Smith, L., Dhawan, N., Zhang, M., Abbeel, P., and Levine, S. ().
Avid: Learning multi-stage tasks via pixel-level translation of human videos.
ArXiv
,abs/.
.Sohn et al.
Sohn, A., Nagabandi, A., Florensa, C., Adelberg, D., Wu, D., Farooq, H., Clavera, I., Welborn, J., Chen, J., Mishra, N., Chen, P., Qian, P., Abbeel, P., Duan, R., Vijay, V., and Liu, Y. ().
Introducing rfm-1: Giving robots human-like reasoning capabilities.
https://covariant.ai/insights/introducing-rfm-1-giving-robots-human-like-reasoning-capabilities/.Accessed: -03-29.
Song et al.
Song, W., Zhao, H., Ding, P., Cui, C., Lyu, S., Fan, Y., and Wang, D. ().
Germ: A generalist robotic model with mixture-of-experts for quadruped robot..
Sontakke et al.
Sontakke, S. A., Zhang, J., Arnold, S. M. R., Pertsch, K., Biyik, E., Sadigh, D., Finn, C., and Itti, L. ().
Roboclip: One demonstration is enough to learn robot policies.
ArXiv
,abs/.
.Srivastava et al.
Srivastava, N., Mansimov, E., and Salakhudinov, R. ().
Unsupervised learning of video representations using lstms.
In
International conference on machine learning
, pp. 843852. PMLR.Stadie et al.
Stadie, B. C., Abbeel, P., and Sutskever, I. ().
Third-person imitation learning..
Stroud et al.
Stroud, J. C., Lu, Z., Sun, C., Deng, J., Sukthankar, R., Schmid, C., and Ross, D. A. ().
Learning video representations from textual web supervision..
Struckmeier and Kyrki
Struckmeier, O., and Kyrki, V. ().
Preventing mode collapse when imitating latent policies from observations..
Sutton
Sutton, R. ().
The bitter lesson..
Sutton
Sutton, R. S. ().
Integrated architectures for learning, planning, and reacting based on approximating dynamic programming.
In
Machine learning proceedings
, pp. 216224. Elsevier.Sutton and Barto
Sutton, R. S., and Barto, A. G. ().
Reinforcement learning: An introduction
.MIT press.
Tam et al.
Tam, A. C., Rabinowitz, N. C., Lampinen, A. K., Roy, N. A., Chan, S. C. Y., Strouse, D., Wang, J. X., Banino, A., and Hill, F. ().
Semantic exploration from language abstractions and pretrained representations.
ArXiv
,abs/.
.Tang et al.
Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., Zhu, R., et al. ().
Video understanding with large language models: A survey..
Tassa et al.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. ().
Deepmind control suite..
Team et al.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. ().
Gemini: a family of highly capable multimodal models..
Team et al.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. ().
Gemma: Open models based on gemini research and technology..
Team et al.
Team, O. M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Xu, C., Luo, J., et al. ().
Octo: An open-source generalist robot policy..
Templeton et al.
Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and Henighan, T. ().
Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet..
Thomas et al.
Thomas, G., Cheng, C.-A., Loynd, R., Vineet, V., Jalobeanu, M., and Kolobov, A. ().
Plex: Making the most of the available data for robotic manipulation pretraining.
ArXiv
,abs/.
.Tirumala et al.
Tirumala, D., Galashov, A., Noh, H., Hasenclever, L., Pascanu, R., Schwarz, J., Desjardins, G., Czarnecki, W. M., Ahuja, A., Teh, Y. W., et al. ().
Behavior priors for efficient reinforcement learning.
Journal of Machine Learning Research
,23
(221), 168.Tobin et al.
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. ().
Domain randomization for transferring deep neural networks from simulation to the real world..
Todorov et al.
Todorov, E., Erez, T., and Tassa, Y. ().
Mujoco: A physics engine for model-based control.
In
IEEE/RSJ international conference on intelligent robots and systems
, pp. . IEEE.Tomar et al.
Tomar, M., Ghosh, D., Myers, V., Dragan, A., Taylor, M. E., Bachman, P., and Levine, S. ().
Video-guided skill discovery.
In
ICML Workshop The Many Facets of Preference-Based Learning
.Tong et al.
Tong, Z., Song, Y., Wang, J., and Wang, L. ().
Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.
ArXiv
,abs/.
.Torabi et al.
Torabi, F., Warnell, G., and Stone, P. (a).
Behavioral cloning from observation.
In
International Joint Conference on Artificial Intelligence
.Torabi et al.
Torabi, F., Warnell, G., and Stone, P. (b).
Generative adversarial imitation from observation.
ArXiv
,abs/.
.Torabi et al.
Torabi, F., Warnell, G., and Stone, P. ().
Recent advances in imitation learning from observation.
In
International Joint Conference on Artificial Intelligence
.Touvron et al.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. ().
Llama 2: Open foundation and fine-tuned chat models..
Tulyakov et al.
Tulyakov, S., Liu, M.-Y., Yang, X., and Kautz, J. ().
Mocogan: Decomposing motion and content for video generation.
In
Proceedings of the IEEE conference on computer vision and pattern recognition
, pp. .van den Oord et al.
van den Oord, A., Vinyals, O., and Kavukcuoglu, K. ().
Neural discrete representation learning.
ArXiv
,abs/.
.Vaswani et al.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. ().
Attention is all you need.
Advances in neural information processing systems
,30
.Villegas et al.
Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., and Erhan, D. ().
Phenaki: Variable length video generation from open domain textual description.
ArXiv
,abs/.
.Villegas et al.
Villegas, R., Pathak, A., Kannan, H., Erhan, D., Le, Q. V., and Lee, H. ().
High fidelity video prediction with large stochastic recurrent neural networks.
Advances in Neural Information Processing Systems
,32
.Vondrick et al.
Vondrick, C., Pirsiavash, H., and Torralba, A. ().
Generating videos with scene dynamics.
Advances in neural information processing systems
,29
.Wang et al.
Wang, C., Luo, X., Ross, K. W., and Li, D. ().
Vrl3: A data-driven framework for visual deep reinforcement learning.
ArXiv
,abs/.
.Wang et al.
Wang, C., Fan, L. J., Sun, J., Zhang, R., Fei-Fei, L., Xu, D., Zhu, Y., and Anandkumar, A. (a).
Mimicplay: Long-horizon imitation learning by watching human play.
ArXiv
,abs/.
.Wang et al.
Wang, J., Dasari, S., Srirama, M. K., Tulsiani, S., and Gupta, A. (b).
Manipulate by seeing: Creating manipulation controllers from pre-trained representations.
ArXiv
,abs/.
.Wang et al.
Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang, Y., and Qiao, Y. (c).
Videomae v2: Scaling video masked autoencoders with dual masking..
Wang et al.
Wang, R., Chen, D., Wu, Z., Chen, Y., Dai, X., Liu, M., Yuan, L., and Jiang, Y.-G. ().
Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning..
Wang et al.
Wang, T., Torralba, A., Isola, P., and Zhang, A. ().
Optimal goal-reaching reinforcement learning via quasimetric learning.
In
International Conference on Machine Learning
, pp. . PMLR.Wang et al.
Wang, X., Yuan, H., Zhang, S., Chen, D., Wang, J., Zhang, Y., Shen, Y., Zhao, D., and Zhou, J. ().
Videocomposer: Compositional video synthesis with motion controllability.
Advances in Neural Information Processing Systems
,36
.Wang et al.
Wang, X., Zhu, Z., Huang, G., Chen, X., and Lu, J. (a).
Drivedreamer: Towards real-world-driven world models for autonomous driving..
Wang et al.
Wang, Y., He, Y., Li, Y., Li, K., Yu, J., Ma, X. J., Chen, X., Wang, Y., Luo, P., Liu, Z., Wang, Y., Wang, L., and Qiao, Y. (b).
Internvid: A large-scale video-text dataset for multimodal understanding and generation.
ArXiv
,abs/.
.Wang et al.
Wang, Z., Hunt, J. J., and Zhou, M. ().
Diffusion policies as an expressive policy class for offline reinforcement learning.
ArXiv
,abs/.
.Wayve
Wayve ().
Lingo-1: Exploring natural language for autonomous driving.
https://wayve.ai/thinking/lingo-natural-language-autonomous-driving/.Accessed: -04-04.
Weissenborn et al.
Weissenborn, D., Täckström, O., and Uszkoreit, J. ().
Scaling autoregressive video models..
Wen et al.
Wen, C., Lin, X., So, J., Chen, K., Dou, Q., Gao, Y., and Abbeel, P. ().
Any-point trajectory modeling for policy learning.
ArXiv
,abs/.
.Whitney et al.
Whitney, W. F., Lopez-Guevara, T., Pfaff, T., Rubanova, Y., Kipf, T., Stachenfeld, K., and Allen, K. R. ().
Learning 3d particle-based simulators from rgb-d videos..
Wu et al.
Wu, B., Nair, S., Martin-Martin, R., Fei-Fei, L., and Finn, C. ().
Greedy hierarchical variational autoencoders for large-scale video prediction.
In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
, pp. .Wu et al.
Wu, H., Jing, Y., Cheang, C.-H., Chen, G., Xu, J., Li, X., Liu, M., Li, H., and Kong, T. (a).
Unleashing large-scale video generative pre-training for visual robot manipulation.
ArXiv
,abs/.
.Wu et al.
Wu, J., Ma, H., Deng, C., and Long, M. (b).
Pre-training contextualized world models with in-the-wild videos for reinforcement learning.
ArXiv
,abs/.
.Wu et al.
Wu, Z., Dvornik, N., Greff, K., Kipf, T., and Garg, A. ().
Slotformer: Unsupervised visual dynamics simulation with object-centric models..
Wulfmeier et al.
Wulfmeier, M., Abdolmaleki, A., Hafner, R., Springenberg, J. T., Neunert, M., Hertweck, T., Lampe, T., Siegel, N., Heess, N. M. O., and Riedmiller, M. A. ().
Compositional transfer in hierarchical reinforcement learning..
Wulfmeier et al.
Wulfmeier, M., Byravan, A., Bechtle, S., Hausman, K., and Heess, N. ().
Foundations for transfer in reinforcement learning: A taxonomy of knowledge modalities..
Xian et al.
Xian, Z., Gervet, T., Xu, Z., Qiao, Y.-L., Wang, T.-H., and Wang, Y. ().
Towards generalist robots: A promising paradigm via generative simulation..
Xie et al.
Xie, A., Lee, L., Xiao, T., and Finn, C. ().
Decomposing the generalization gap in imitation learning for visual robotic manipulation..
Xing et al.
Xing, J., Xia, M., Liu, Y., Zhang, Y., He, Y., Liu, H., Chen, H., Cun, X., Wang, X., Shan, Y., et al. ().
Make-your-video: Customized video generation using textual and structural guidance
Xiong et al.
Xiong, H., Fu, H., Zhang, J., Bao, C., Zhang, Q., Huang, Y., Xu, W., Garg, A., and Lu, C. ().
Robotube: Learning household manipulation from human videos with simulated twin environments.
In
Conference on Robot Learning
.Xiong et al.
Xiong, H., Li, Q., Chen, Y.-C., Bharadhwaj, H., Sinha, S., and Garg, A. ().
Learning by watching: Physical imitation of manipulation skills from human videos..
Xu et al.
Xu, H., Zhang, J., Cai, J., Rezatofighi, H., and Tao, D. ().
Gmflow: Learning optical flow via global matching.
In
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
, pp. .Xu et al.
Xu, H., Ghosh, G., Huang, P.-Y. B., Okhonko, D., Aghajanyan, A., and Feichtenhofer, F. M. L. Z. C. ().
Videoclip: Contrastive pre-training for zero-shot video-text understanding.
In
Conference on Empirical Methods in Natural Language Processing
.Xu et al.
Xu, M., Xu, Z., Chi, C., Veloso, M. M., and Song, S. ().
Xskill: Cross embodiment skill discovery.
ArXiv
,abs/.
.Xu et al.
Xu, M., Dai, W., Liu, C., Gao, X., Lin, W., Qi, G.-J., and Xiong, H. ().
Spatial-temporal transformer networks for traffic flow forecasting.
ArXiv
,abs/.
.Xue et al.
Xue, H., Hang, T., Zeng, Y., Sun, Y., Liu, B., Yang, H., Fu, J., and Guo, B. ().
Advancing high-resolution video-language representation with large-scale video transcriptions.
In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
, pp. .Yan et al.
Yan, S., Zhu, T., Wang, Z., Cao, Y., Zhang, M., Ghosh, S., Wu, Y., and Yu, J. ().
Videococa: Video-text modeling with zero-shot transfer from contrastive captioners..
Yan et al.
Yan, W., Hafner, D., James, S., and Abbeel, P. ().
Temporally consistent transformers for video generation.
In
International Conference on Machine Learning
, pp. . PMLR.Yan et al.
Yan, W., Zhang, Y., Abbeel, P., and Srinivas, A. ().
Videogpt: Video generation using vq-vae and transformers.
ArXiv
,abs/.
.Yang et al.
Yang, J., Mark, M. S., Vu, B., Sharma, A., Bohg, J., and Finn, C. (a).
Robot fine-tuning made easy: Pre-training rewards and policies for autonomous real-world reinforcement learning..
Yang et al.
Yang, M., Du, Y., Dai, B., Schuurmans, D., Tenenbaum, J. B., and Abbeel, P. (b).
Probabilistic adaptation of text-to-video models.
ArXiv
,abs/.
.Yang et al.
Yang, M., Du, Y., Ghasemipour, K., Tompson, J., Schuurmans, D., and Abbeel, P. (c).
Learning interactive real-world simulators.
ArXiv
,abs/.
.Yang et al.
Yang, M., Schuurmans, D., Abbeel, P., and Nachum, O. ().
Dichotomy of control: Separating what you can control from what you cannot.
ArXiv
,abs/.
.Yang et al.
Yang, S., Nachum, O., Du, Y., Wei, J., Abbeel, P., and Schuurmans, D. ().
Foundation models for decision making: Problems, methods, and opportunities.
ArXiv
,abs/.
.Yang et al.
Yang, S., Walker, J., Parker-Holder, J., Du, Y., Bruce, J., Barreto, A., Abbeel, P., and Schuurmans, D. ().
Video as the new language for real-world decision making..
Yang et al.
Yang, T., Zhu, Y., Xie, Y., Zhang, A., Chen, C., and Li, M. ().
Aim: Adapting image models for efficient video action recognition.
ArXiv
,abs/.
.Yao et al.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. ().
React: Synergizing reasoning and acting in language models..
Yarats et al.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. ().
Mastering visual continuous control: Improved data-augmented reinforcement learning.
ArXiv
,abs/.
.Yarats et al.
Yarats, D., Zhang, A., Kostrikov, I., Amos, B., Pineau, J., and Fergus, R. ().
Improving sample efficiency in model-free reinforcement learning from images.
In
AAAI Conference on Artificial Intelligence
.Ye et al.
Ye, W., Zhang, Y., Wang, M., Wang, S., Gu, X., Abbeel, P., and Gao, Y. ().
Foundation reinforcement learning: towards embodied generalist agents with foundation prior assistance.
ArXiv
,abs/.
.Young et al.
Young, S., Gandhi, D., Tulsiani, S., Gupta, A. K., Abbeel, P., and Pinto, L. ().
Visual imitation made easy.
In
Conference on Robot Learning
.Yu et al.
Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M.-H., Hao, Y., Essa, I., and Jiang, L. ().
Magvit: Masked generative video transformer..
Yu et al.
Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D. C., Cheng, Y., Gupta, A., Gu, X., Hauptmann, A. G., Gong, B., Yang, M.-H., Essa, I., Ross, D. A., and Jiang, L. ().
Language model beats diffusion tokenizer is key to visual generation..
Yu et al.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. ().
Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.
In
Conference on robot learning
, pp. . PMLR.Yu et al.
Yu, T., Xiao, T., Stone, A., Tompson, J., Brohan, A., Wang, S., Singh, J., Tan, C., Peralta, J., Ichter, B., et al. (a).
Scaling robot learning with semantically imagined experience..
Yu et al.
Yu, W., Gileadi, N., Fu, C., Kirmani, S., Lee, K.-H., Arenas, M. G., Chiang, H.-T. L., Erez, T., Hasenclever, L., Humplik, J., et al. (b).
Language to rewards for robotic skill synthesis..
Yuan et al.
Yuan, C., Wen, C., Zhang, T., and Gao, Y. ().
General flow as foundation affordance for scalable robot learning.
ArXiv
,abs/.
.Yuan et al.
Yuan, H., Wu, R., Zhao, A., Zhang, H., Ding, Z., and Dong, H. ().
Dmotion: Robotic visuomotor control with unsupervised forward model learned from videos..
Zakka et al.
Zakka, K., Zeng, A., Florence, P. R., Tompson, J., Bohg, J., and Dwibedi, D. ().
Xirl: Cross-embodiment inverse reinforcement learning.
In
Conference on Robot Learning
.Zellers et al.
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., and Choi, Y. ().
Merlot: Multimodal neural script knowledge models.
Advances in Neural Information Processing Systems
,34
, .Zeng et al.
Zeng, A., Attarian, M., Ichter, B., Choromanski, K., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., et al. ().
Socratic models: Composing zero-shot multimodal reasoning with language..
Zhang et al.
Zhang, D. J., Wu, J. Z., Liu, J.-W., Zhao, R., Ran, L., Gu, Y., Gao, D., and Shou, M. Z. ().
Show-1: Marrying pixel and latent diffusion models for text-to-video generation..
Zhang et al.
Zhang, D., Yu, Y., Li, C., Dong, J., Su, D., Chu, C., and Yu, D. ().
Mm-llms: Recent advances in multimodal large language models..
Zhang et al.
Zhang, H., Li, X., and Bing, L. ().
Video-llama: An instruction-tuned audio-visual language model for video understanding..
Zhang et al.
Zhang, L., Zhou, S., Stent, S., and Shi, J. ().
Fine-grained egocentric hand-object segmentation: Dataset, model, and applications.
In
European Conference on Computer Vision
, pp. 127145. Springer.Zhao et al.
Zhao, L., Gundavarapu, N. B., Yuan, L., Zhou, H., Yan, S., Sun, J. J., Friedman, L., Qian, R., Weyand, T., Zhao, Y., Hornung, R., Schroff, F., Yang, M., Ross, D. A., Wang, H., Adam, H., Sirotenko, M., Liu, T., and Gong, B. ().
Videoprism: A foundational visual encoder for video understanding..
Zhao et al.
Zhao, T., Karamcheti, S., Kollar, T., Finn, C., and Liang, P. ().
What makes representation learning from videos hard for control?..
Zhao et al.
Zhao, W., Queralta, J. P., and Westerlund, T. ().
Sim-to-real transfer in deep reinforcement learning for robotics: a survey.
In
IEEE symposium series on computational intelligence (SSCI)
, pp. 737744. IEEE.Zhen et al.
Zhen, H., Qiu, X., Chen, P., Yang, J., Yan, X., Du, Y., Hong, Y., and Gan, C. ().
3d-vla: A 3d vision-language-action generative world model..
Zhou et al.
Zhou, D., Wang, W., Yan, H., Lv, W., Zhu, Y., and Feng, J. ().
Magicvideo: Efficient video generation with latent diffusion models..
Zhou et al.
Zhou, S., Du, Y., Chen, J., Li, Y., Yeung, D.-Y., and Gan, C. ().
Robodreamer: Learning compositional world models for robot imagination..
Zhou et al.
Zhou, T., Tucker, R., Flynn, J., Fyffe, G., and Snavely, N. ().
Stereo magnification.
ACM Transactions on Graphics (TOG)
,37
, 1 12.Zhou et al.
Zhou, W., Bajracharya, S., and Held, D. ().
Plas: Latent action space for offline reinforcement learning.
In
Conference on Robot Learning
, pp. . PMLR.Zhu et al.
Zhu, D., Wang, Y., Schmidhuber, J., and Elhoseiny, M. ().
Guiding online reinforcement learning with action-free offline pretraining.
ArXiv
,abs/.
.Zhuang et al.
Zhuang, Z., Fu, Z., Wang, J., Atkeson, C., Schwertfeger, S., Finn, C., and Zhao, H. ().
Robot parkour learning..
Zou et al.
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. ().
Representation engineering: A top-down approach to ai transparency..
Want more information on FUXIN Robot? Feel free to contact us.