Troubleshooting High I/O Wait in Linux

Photo by Eutah Mizushima on Unsplash

Answering whether or not I/O is causing system slowness

We can use several commands to identify whether I/O is causing system slowness, but the easiest is the Unix command top.

Finding which disk is being written to

In the example above, the top command shows I/O Wait from the system as a whole. But it does not tell us what disk is being affected; we will use the iostat command.

Finding the processes that are causing high I/O

A great tool for identifying processes accessing disk is iotop.

Process list “state”

The ps command is one many will be familiar with, but few will realize has statistics for memory and CPU usage.

  • D: uninterruptible sleep (usually IO)
  • R: running or runnable (on run queue)
  • S: interruptible sleep (waiting for an event to complete)
  • T: stopped, either by a job control signal or because it is being traced
  • W: paging (not valid since the 2.6.xx kernel)
  • X: dead (should never be seen)
  • Z: defunct (“zombie”) process, terminated but not reaped by its parent

Finding what files are being written too heavily

Once the process is identified, or a set of processes are suspected. We can use the lsof command to see what files the specific processes have open. This list of open files will allow us to make an educated guess as to what files are being written to most often.

More Linux Troubleshooting Guides

For more practical Linux Troubleshooting examples check out my book, Red Hat Enterprise Linux Troubleshooting Guide.

--

--

Building resilient & performant payments systems at American Express. Author (https://amzn.to/3kuCFpz). Open-source contributor. Thoughts and opinions are mine.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Benjamin Cane

Building resilient & performant payments systems at American Express. Author (https://amzn.to/3kuCFpz). Open-source contributor. Thoughts and opinions are mine.