Machine Learning

---

Positional Normalization (1907.04312v1)

Boyi Li, Felix Wu, Kilian Q. Weinberger, Serge Belongie

2019-07-09

A widely deployed method for reducing the training time of deep neural networks is to normalize activations at each layer. Although various normalization schemes have been proposed, they all follow a common theme: normalize across spatial dimensions and discard the extracted statistics. In this paper, we propose a novel normalization method that noticeably departs from this convention. Our approach, which we refer to as Positional Normalization (PONO), normalizes exclusively across channels --- a naturally appealing dimension, which captures the first and second moments of features extracted at a particular image position. We argue that these moments convey structural information about the input image and the extracted features, which opens a new avenue along which a network can benefit from feature normalization: Instead of disregarding the PONO normalization constants, we propose to re-inject them into later layers to preserve or transfer structural information in generative networks.

Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering (1907.04298v1)

Pedro F. Proenca, Yang Gao

2019-07-09

On-orbit proximity operations in space rendezvous, docking and debris removal require precise and robust 6D pose estimation under a wide range of lighting conditions and against highly textured background, i.e., the Earth. This paper investigates leveraging deep learning and photorealistic rendering for monocular pose estimation of known uncooperative spacecrafts. We first present a simulator built on Unreal Engine 4, named URSO, to generate labeled images of spacecrafts orbiting the Earth, which can be used to train and evaluate neural networks. Secondly, we propose a deep learning framework for pose estimation based on orientation soft classification, which allows modelling orientation ambiguity as a mixture of Gaussians. This framework was evaluated both on URSO datasets and the ESA pose estimation challenge. In this competition, our best model achieved 3rd place on the synthetic test set and 2nd place on the real test set. Moreover, our results show the impact of several architectural and training aspects, and we demonstrate qualitatively how models learned on URSO datasets can perform on real images from space.

Pixel-Attentive Policy Gradient for Multi-Fingered Grasping in Cluttered Scenes (1903.03227v3)

Bohan Wu, Iretiayo Akinola, Peter K. Allen

2019-03-08

Recent advances in on-policy reinforcement learning (RL) methods enabled learning agents in virtual environments to master complex tasks with high-dimensional and continuous observation and action spaces. However, leveraging this family of algorithms in multi-fingered robotic grasping remains a challenge due to large sim-to-real fidelity gaps and the high sample complexity of on-policy RL algorithms. This work aims to bridge these gaps by first reinforcement-learning a multi-fingered robotic grasping policy in simulation that operates in the pixel space of the input: a single depth image. Using a mapping from pixel space to Cartesian space according to the depth map, this method transfers to the real world with high fidelity and introduces a novel attention mechanism that substantially improves grasp success rate in cluttered environments. Finally, the direct-generative nature of this method allows learning of multi-fingered grasps that have flexible end-effector positions, orientations and rotations, as well as all degrees of freedom of the hand.

UW-BHI at MEDIQA 2019: An Analysis of Representation Methods for Medical Natural Language Inference (1907.04286v1)

William R. Kearns, Wilson Lau, Jason A. Thomas

2019-07-09

Recent advances in distributed language modeling have led to large performance increases on a variety of natural language processing (NLP) tasks. However, it is not well understood how these methods may be augmented by knowledge-based approaches. This paper compares the performance and internal representation of an Enhanced Sequential Inference Model (ESIM) between three experimental conditions based on the representation method: Bidirectional Encoder Representations from Transformers (BERT), Embeddings of Semantic Predications (ESP), or Cui2Vec. The methods were evaluated on the Medical Natural Language Inference (MedNLI) subtask of the MEDIQA 2019 shared task. This task relied heavily on semantic understanding and thus served as a suitable evaluation set for the comparison of these representation methods.

Deep Learning Techniques for Improving Digital Gait Segmentation (1907.04281v1)

Matteo Gadaleta, Giulia Cisotto, Michele Rossi, Rana Zia Ur Rehman, Lynn Rochester, Silvia Del Din

2019-07-09

Wearable technology for the automatic detection of gait events has recently gained growing interest, enabling advanced analyses that were previously limited to specialist centres and equipment (e.g., instrumented walkway). In this study, we present a novel method based on dilated convolutions for an accurate detection of gait events (initial and final foot contacts) from wearable inertial sensors. A rich dataset has been used to validate the method, featuring 71 people with Parkinson's disease (PD) and 67 healthy control subjects. Multiple sensors have been considered, one located on the fifth lumbar vertebrae and two on the ankles. The aims of this study were: (i) to apply deep learning (DL) techniques on wearable sensor data for gait segmentation and quantification in older adults and in people with PD; (ii) to validate the proposed technique for measuring gait against traditional gold standard laboratory reference and a widely used algorithm based on wavelet transforms (WT); (iii) to assess the performance of DL methods in assessing high-level gait characteristics, with focus on stride, stance and swing related features. The results showed a high reliability of the proposed approach, which achieves temporal errors considerably smaller than WT, in particular for the detection of final contacts, with an inter-quartile range below 70 ms in the worst case. This study showes encouraging results, and paves the road for further research, addressing the effectiveness and the generalization of data-driven learning systems for accurate event detection in challenging conditions.

A Conformance Checking-based Approach for Drift Detection in Business Processes (1907.04276v1)

Víctor Gallego-Fontenla, Juan C. Vidal, Manuel Lama

2019-07-09

Real life business processes change over time, in both planned and unexpected ways. The detection of these changes is crucial for organizations to ensure that the expected and the real behavior are as similar as possible. These changes over time are called concept drift and its detection is a big challenge in process mining since the inherent complexity of the data makes difficult distinguishing between a change and an anomalous execution. In this paper, we present C2D2 (Conformance Checking-based Drift Detection), a new approach to detect sudden control-flow changes in the process models from event traces. C2D2 combines discovery techniques with conformance checking methods to perform an offline detection. Our approach has been validated with a synthetic benchmarking dataset formed by 68 logs, showing an improvement in the accuracy while maintaining a minimum delay in the drift detection.

Learning to Optimize Domain Specific Normalization for Domain Generalization (1907.04275v1)

Seonguk Seo, Yumin Suh, Dongwan Kim, Jongwoo Han, Bohyung Han

2019-07-09

We propose a simple but effective multi-source domain generalization technique based on deep neural networks by incorporating optimized normalization layers specific to individual domains. Our key idea is to decompose discriminative representations in each domain into domain-agnostic and domain-specific components by learning a mixture of multiple normalization types. Because each domain has different characteristics, we optimize the mixture weights specialized to each domain and maximize the generalizability of the learned representations per domain. To this end, we incorporate instance normalization into the network with batch normalization since instance normalization is effective to discard the discriminative domain-specific representations. Since the joint optimization of the parameters in convolutional and normalization layers is not straightforward especially in the lower layers, the mixture weight of the normalization types is shared across all layers for the robustness of trained models. We analyze the effectiveness of the optimized normalization layers and demonstrate the state-of-the-art accuracy of our algorithm in the standard benchmark datasets in various settings.

On orthogonal projections for dimension reduction and applications in augmented target loss functions for learning problems (1901.07598v3)

Anna Breger, Jose Ignacio Orlando, Pavol Harar, Monika Dörfler, Sophie Klimscha, Christoph Grechenig, Bianca S. Gerendas, Ursula Schmidt-Erfurth, Martin Ehler

2019-01-22

The use of orthogonal projections on high-dimensional input and target data in learning frameworks is studied. First, we investigate the relations between two standard objectives in dimension reduction, preservation of variance and of pairwise relative distances. Investigations of their asymptotic correlation as well as numerical experiments show that a projection does usually not satisfy both objectives at once. In a standard classification problem we determine projections on the input data that balance the objectives and compare subsequent results. Next, we extend our application of orthogonal projections to deep learning tasks and introduce a general framework of augmented target loss functions. These loss functions integrate additional information via transformations and projections of the target data. In two supervised learning problems, clinical image segmentation and music information classification, the application of our proposed augmented target loss functions increase the accuracy.

A divide-and-conquer algorithm for binary matrix completion (1907.04251v1)

Melanie Beckerleg, Andrew Thompson

2019-07-09

We propose an algorithm for low rank matrix completion for matrices with binary entries which obtains explicit binary factors. Our algorithm, which we call TBMC (\emph{Tiling for Binary Matrix Completion}), gives interpretable output in the form of binary factors which represent a decomposition of the matrix into tiles. Our approach is inspired by a popular algorithm from the data mining community called PROXIMUS: it adopts the same recursive partitioning approach while extending to missing data. The algorithm relies upon rank-one approximations of incomplete binary matrices, and we propose a linear programming (LP) approach for solving this subproblem. We also prove a -approximation result for the LP approach which holds for any level of subsampling and for any subsampling pattern. Our numerical experiments show that TBMC outperforms existing methods on recommender systems arising in the context of real datasets.

Effect of Depth and Width on Local Minima in Deep Learning (1811.08150v4)

Kenji Kawaguchi, Jiaoyang Huang, Leslie Pack Kaelbling

2018-11-20

In this paper, we analyze the effects of depth and width on the quality of local minima, without strong over-parameterization and simplification assumptions in the literature. Without any simplification assumption, for deep nonlinear neural networks with the squared loss, we theoretically show that the quality of local minima tends to improve towards the global minimum value as depth and width increase. Furthermore, with a locally-induced structure on deep nonlinear neural networks, the values of local minima of neural networks are theoretically proven to be no worse than the globally optimal values of corresponding classical machine learning models. We empirically support our theoretical observation with a synthetic dataset as well as MNIST, CIFAR-10 and SVHN datasets. When compared to previous studies with strong over-parameterization assumptions, the results in this paper do not require over-parameterization, and instead show the gradual effects of over-parameterization as consequences of general results.

---