An isosceles right triangular waveguide with one electric wall and two magnetic walls is proposed to design a triple-mode filter implemented in substrate integrated waveguide. The complete closed-form modal solutions are presented for both transverse electric and transverse magnetic modes in the isosceles right triangular waveguide. The resonant frequencies of a resonator made of a short isosceles right triangular waveguide can then be calculated. A structure named as quarter-mode substrate integrated waveguide (QMSIW) is introduced to realize the proposed triangular waveguide resonator in a compact and planar form. Simulated electric field distributions of modes excited in the QMSIW are in good agreement with theoretical predictions for the solid waveguide structure, demonstrating the feasibility of QMSIW resonators. Finally, a compact triple-mode filter is designed and fabricated based on the discussed QMSIW. The frequency responses and group delay of the filter are tested, and measured results agree very well with simulated ones. This demonstrates that the proposed QMSIW structure is an excellent candidate for compact triple-mode filters.