Last week I installed a fresh WSUS server for a customer of mine and because it needed to download lots of files after the approvals were done we left it for a few days. Today I came in and opened the WSUS console only to notice it refused to connect. Got an error like this one:
The WSUS administration console was unable to connect to the WSUS Server via the remote API.
Verify that the Update Services service, IIS and SQL are running on the server. If the problem persists, try restarting IIS, SQL, and the Update Services Service.
The WSUS administration console has encountered an unexpected error. This may be a transient error; try restarting the administration console. If this error persists, Try removing the persisted preferences for the console by deleting the wsus file under %appdata%\Microsoft\MMC\.
System.IO.IOException — The handshake failed due to an unexpected packet format.
After checking that the requires services were running the investigation starts. Lot of blog and forum posts from long ago to recent, all with different solutions.
I came across a post from 6 weeks or so ago which talks about an update KB3148812 which causes this behavior and also to cause an additional error where clients can not scan WSUS.
Now I could not find this KB patch installed on my system, however it mentioned manual steps to be done after applying the hotfix and those manual steps solved it indeed. Keep reading.
A little more research found that the 3148812 has now been cancelled and another one came in its place KB3159706.
This article describes what is going on and it contains manual steps to be followed! The first step solved the console not being able to connect. The second step is for HTTP Activation. And if you have SSL turned on there are a few more steps to follow.