Boot the device into Safe Mode and see if the issue persists.
Hold your hand over the sensor, and the value should switch between 5 cm and 0 cm for far and near respectively. Go to the Sensors section and look for the Proximity sensor.
Download an app like Aida64 from the Google Play Store. You can also check if there is a problem with the proximity sensor itself.Screen guards made specifically for the Nexus 5X come with an L shaped cut at the top that keep the proximity sensor uncovered, and using this helps avoid the problem. This seems to be a problem with the proximity sensor, and is generally faced by users that use generic screen guards, especially of the tempered glass variety.
