In this dissertation, we consider models with low-rank and group-sparse components. First, we investigate robust principal component analysis, where the low-rank component represents the principal components, and the group-sparse component accounts for corruptions in the data. We propose a model for the general setting, where groups of observed variables can be corrupted. Second, we generalize fused latent and graphical models to the class of conditional Gaussian distributions with mixed observed discrete and quantitative variables. Fused latent and graphical models are characterized by a decomposition of the pairwise interaction parameter matrix into a group-sparse component of direct interactions and a low-rank component of indirect interactions due to a small number of quantitative latent variables. All models in this thesis can be learned by solving convex optimization problems with low-rank and group-sparsity inducing regularization terms. For fused latent and graphical models, there is an additional likelihood term. We show that under identifiability assumptions, a given true model can be recovered exactly (principal component analysis) or consistently (fused latent and graphical models, high-dimensional setting) by solving the respective optimization problems. We also present heuristics for selecting the regularization parameters that appear in the optimization problems. We conduct experiments on synthetic and real-world data to support our theory.