A miniaturized optical immunosensor based on White Light Reflectance Spectroscopy (WLRS) for the rapid and label-free detection of aflatoxin M1 (AFM1) in milk samples has been developed. WLRS sensing system consists of the measurement set-up and the biochip. The first encompasses the reflection probe, the light source, and the spectrometer, while the latter is a Si chip with a SiO2 layer on top where an AFM1-bovine serum albumin conjugate has been immobilized. The assay was performed by running mixtures of rabbit polyclonal anti-AFM1 antibody with the calibrators or the samples, followed by reaction with biotinylated anti-rabbit IgG antibody and streptavidin. The assay cycle was completed in 25 min, the limit of detection was 6 pg/mL, and the linear working range extended from 0.012 to 2.0 ng/mL. The assay was repeatable (intra-and inter-assay coefficients of variation ranged from 2.1 to 6.3% and 3.5 to 8.2%, respectively) and accurate (percent recovery values ranged from 92.5 to 110%). AFM1 could be detected with the immunosensor developed in both processed and unprocessed milk of different animal species without any dilution. The excellent analytical characteristics and the small instrument size make the proposed sensor suitable for accurate low-cost AFM1 determination in milk samples at the point-of-need.