World number 81 Poulter has played six tournaments in seven weeks but must now rely on a rankings boost if he is to make it to Erin Hills.
Poulter needs a high finish at next week’s FedEX St Jude Classic to secure the top-60 ranking that will secure his place in the U.S. Open.
Failure to qualify for the second major of the year would see Poulter switch his attention to a bid to reach The Open at Royal Birkdale in July.
“I felt it might be just a little bit too much golf,” Poulter said of his decision to withdraw from the qualifier.
“A 36-hole qualifier on Monday in Memphis could be a little much leading in.”
The U.S. Open gets under way in Erin, Wisconsin, on June 15.