The present summary of publications outlines several theoretical and numerical investigations of instabilities and complex nonlinear dynamics in fluid flows. The summary mostly presents the motivation of the different studies and the essential physics of the systems under consideration. Main results are also described. Details of the mathematical models and numerical methods are given in the publications.The contributions of the author to the individual papers are described in a separate appendix. The flows under study are thermal convection in single and immiscible double fluid layers, two-phase shear layers, and channel flows of electrically conducting fluid with an imposed magnetic field. These configurations are motivated by a variety of applications, e.g. interfacial heat and mass transferin chemical engineering processes, liquid atomization for combustion, or electromagnetic pumps and brakes for the processing of materials in metallurgy. The complexity of the real applications has been reduced considerably in order to examine fundamental mechanisms and properties. The geometric and conceptual simplicity achieved this way is also useful for the numerical studies since it typically allows one to use specialized but very efficient simulation methods. The results of such investigations can improve our understanding of flow physics and can also serve as benchmarks for the verification of more general computational approaches. Thermal convection was considered in several configurations.The first is purely surface-tension driven convection in a single liquid layer, for which the flow structure and the heat flux scaling was studied by two-dimensional and three-dimensional simulations.The second is a system of two layers with heating from below or from aboveand different combinations of immiscible liquids. Two different two-layer setups were studied by three-dimensional numerical simulations in the nonlinear regime with a focus on the transformation of the convective patterns with the thermal forcing. On the topic of two-phase mixing layers two linear stability studies based on coupled Rayleigh/Orr-Sommerfeld equations were performed, and a verification of the nonlinear simulation code SURFER by means of the viscous linear stability results. The novelty in the linear stability calculations consisted in a direct comparison of viscous and inviscid results for geometrically equivalent configurations, and in the identification of a specific viscous instability mechanism in the parameter range of experiments on air/water atomization. Finally, on the topic of channel flows of electrically conducting fluid with an imposed magnetic field the author has been involved in numerical studies of transition and turbulence in conducting channel flows with uniform magnetic field. A nonlinear transition mechanism for subcritical Reynolds numbers was investigated for a spanwise magnetic field, and the properties of magnetohydrodynamic turbulence were studied for both wall-normal and spanwise magnetic field.