Continuous glucose monitors (CGM) and insulin pumps are becoming increasingly important in diabetes man- agement. Additionally, data streams from these devices enable the prospect of accurate blood glucose prediction to support patients in preventing adverse glycemic events. In this paper, we present Neural Physiological Encoder (NPE), a simple module that leverages decomposed convolutional filters to automatically generate effective features that can be used with a downstream neural network for blood glucose prediction. To our knowledge, this is the first work to investigate a decomposed architecture in the diabetes domain. Our experimental results show that the proposed NPE model can effectively capture temporal patterns and blood glucose associations with other daily activities. For predicting blood glucose 30-mins in advance, NPE+LSTM yields an average root mean square error (RMSE) of 9.18 mg/dL on an in-house diabetes dataset from 34 subjects. Additionally, it achieves state-of-the-art RMSE of 17.80 mg/dL on a publicly available diabetes dataset (OhioT1DM) from 6 subjects.