Building occupancy estimation holds great promise for building control systems to save energy and provide a comfortable indoor environment. Existing solutions turn out to be lacking in practice due to their specific hardware requirements and/or poor performances. Recently, an LED light sensor based occupancy estimation system which is non-intrusive and does not require any additional hardware has been proposed. However, the performance of the system is limited, especially in a complicated dynamic scenario. In this paper, a Bayes filter with neural networks is proposed for the optimal estimation of occupancy based on light sensor data. Specifically, based on the formulation of Bayes filter, the posterior probability of the building occupancy can be decoupled into three components: the prior, likelihood and evidence. The prior and likelihood are respectively estimated from a Markov model and an efficient Single-hidden Layer Feedforward Neural network (SLFN). Finally, the evidence can be obtained by the results of prior and likelihood. Real experiments have been conducted to verify the effectiveness of the proposed approach in two complicated scenarios, i.e., dynamic and regular. Results indicate that the proposed Bayes filter outperforms all the benchmark approaches. The impacts of the number of LED sensing units and the number of hidden layers for neural networks are also evaluated. The results manifest that the number of sensing units should be chosen based on the required performance and the SLFN is sufficient for this application.