Brownian motion influences bacterial swimming by randomizing displacement and direction. Here, we report that the influence of Brownian motion is amplified when it is coupled to hydrodynamic interaction. We examine swimming trajectories of the singly flagellated bacterium Caulobacter crescentus near a glass surface with total internal reflection fluorescence microscopy and observe large fluctuations over time in the distance of the cell from the solid surface caused by Brownian motion. The observation is compared with computer simulation based on analysis of relevant physical factors, including electrostatics, van der Waals force, hydrodynamics, and Brownian motion. The simulation reproduces the experimental findings and reveals contribution from fluctuations of the cell orientation beyond the resolution of present observation. Coupled with hydrodynamic interaction between the bacterium and the boundary surface, the fluctuations in distance and orientation subsequently lead to variation of the swimming speed and local radius of curvature of swimming trajectory. These results shed light on the fundamental roles of Brownian motion in microbial motility, nutrient uptake, and adhesion.