When Windows Hosts Files Stop Working

So, working on what should have been a quick PowerShell script today, I ran into quite the hiccup. Annoying as hell.

My Hosts file on my Windows 7 Machine stopped working!

This was a rage inducing issue for me. But luckily I think I've identified the "problem." And yes, it is technically PEBKAC - but I think Microsoft should carry a little of the blame too.

First let us set the stage with a very basic PowerShell script to setup the hosts file the way we'd like.

$hosts = @()
$hosts = $hosts + [String]::Format("{0} {1}", "192.168.0.1", "some.local.domain")
$hosts = $hosts + [String]::Format("{0} {1}", "127.0.0.1", "www.somesite.com")
$hosts > $env:windir\system32\drivers\etc\hosts

Note: This must be run with appropriate permissions to write to the hosts file!

Pretty straight forward, make an array, add two strings to it with formatted IP/domain pairs. Write the array out to the file. When this is all said and done you should have something which looks like the following:

This is what we would expect. A hosts file with two entries that says some.local.domain -> 192.168.0.1 and another entry that says www.somesite.com -> 127.0.0.1.

Looks correct to me - and if you're reading this yours probably looks correct to you too. But as you can see the ping commands are not resolving based on the IPs in the hosts file! This is certainly not the desired effect. As if the hosts file were being "skipped."

What's the deal?

The error that resides in the above script does not reveal itself when opening the hosts file in your text editor. And in fact in some text editors it'll exist all of the time. Why you ask?

Simple; Your text editor probably detects and allows you to edit Unicode files. Optionally your text editor only supports Unicode files.

Yep, the whole annoying as hell bug - is just a simple encoding problem.

Here's how I noticed it - I opened my hosts file in a hex editor (HxD in my case, cause it's free). Viewing the raw hex we see:

This thought occurred to me as a mere suspicion. I did what I've seen a number of sources say to do: open the hosts file, copy it's contents, remove the hosts file and then re-save the contents using just plain-ol'e notepad. This does work, but it's not a solution for automation in the case of scripts (like I need it to be).

Also remember notepad will save it with the extensions .txt and you'll need to rename it.

So this works, in theory, because when notepad saves the contents it saves them in ASCII and stripping the Unicode encoding. Which is seen when viewing the contents of the new hosts file in HxD:

So, how does one fix the PowerShell script to work too? You're in luck,  it's an easy fix. Annoyingly easy after so many hours bashing your head against a keyboard over nothing.

Setup your array in the same way, when you're ready just pass your array to Out-File like so:

$hosts | Out-File -Encoding ASCII -FilePath $env:windir\system32\drivers\etc\hosts

And now, you should have similar - but better - results! Notice how notepad looks exactly the same! But the pings resolve the correct names, etc. Thus confirming my hypothesis.

That's it, I hope this helps someone. So as usual....
Hack Legal, Hack Safe, buy most of all Hack Fun!
Until next time...

Powershell and MSDeploy.exe

As some people may know, I've been playing with Powershell a lot for work lately. Most recently I've also been using joining powershell with a Web Deploy executable, msdeploy.exe. And I've run into a couple of bugs I've noticed some other people have also had issues with.

At my place of employment we have a multi-tier development environment. This consists of 3 tiers we adoringly call "dev," "test," and "prod." We also have some lateral environments (lateral to our test environment mostly) which we call "staging" and "training." On to the problem and solution...

The issue was syncing a remote server's IIS AppHostConfig settings to my local machine. I tried to sync our dev environment's settings to a package locally, which I will later sync that package to my localhost environment. This script's goal is to allow us to rebuild a development box in rapid order when a developer gets a new or re-imaged machine. Here's the command I was using:

msdeploy.exe -verb:sync -source:appHostConfig="Site1",computername=devserver -disableLink:ContentExtension -disableLink:CertificateExtension -disableLink:FrameworkConfigExtension -disableLink:HttpCertConfigExtension -dest:package=c:\IIS.zip

This was pretty much stolen right out of Synchronize IIS 7 and was very disappointing to see this error occur:

Error: Unrecognized argument '"-source:appHostConfig=Site1  computername=devserver"'. All arguments must begin with "-".
Error count: 1.

Pretty much my first response was a lipping of  "W.T.F." (we have Mormons in the office who may otherwise be offended if I yelled this top of my lungs) followed by an intense urge to Google the error.

So I did.

I did not find any great results...

Then I realized of course, this is probably because all the information I need is in the error! Unrecognized argument "-source:" So what follows the colon must be the expected argument... which we have "appHostConfig=Site1 computername=devserver" wait a second... That isn't what I typed, there is no space, I used a comma! Where'd it go? I am gonna guess it was stripped out by powershell which then invalidates the whole argument for -source.

The Solution you ask? A little quotation magic:

msdeploy.exe -verb:sync -source:"appHostConfig='Site1',computername=devserver" -disableLink:ContentExtension -disableLink:CertificateExtension -disableLink:FrameworkConfigExtension -disableLink:HttpCertConfigExtension -dest:package=c:\IIS.zip


And presto, the double quotes force the whole inner text to be 1 argument, without stripping out the comma. Which is precisely what msdeploy.exe expects. IIS synced.


Cheers,
  -Ken

Powershell: Start-BitsTransfer & User Authentication

I recently discussed an issue with ActualReverend (alright it was a month and a half ago and I'm just now blogging about it) concerning Powershell and using Start-BitsTransfer against a web server requiring user authentication. Additionally, this is my first time ever using Powershell for any task more than running 'ipconfig /all'. So I suppose we'll start with our problem statement:

"Using 'Start-BitsTransfer', download a file from a web server (an apache/svn service), which requires user authentication against an LDAP server, to a given location. Path is '/path/file.ps1' user credentials are 'username:password.' "

Should be a fairly simple script, he needed it for more but I am mostly interested with why it's not working as expected.

$creds = Get-Credential

Start-Bitstransfer -Source https://ourmixbreedserver.domain.com/path/file.ps1 -Destination C:\destination\ -Credential $creds -Authentication basic

Pretty straight forward script. It prompts the user for their credentials we enter "username" and "password" as that's our credentials. The authentication type is basic, and the server is properly configured to goto LDAP, so we're ready to Rock 'N Roll... But the results are of course not as expected, let's see the error.

Start-BitsTransfer : HTTP status 401: The requested resource requires user authentication.

Well, damn. We thought that providing the username and password pair would cover us but something went wrong and the error is not as forthcoming with the cause as we'd like. We provided credentials and the fact that we wanted the cmdlet to use authentication, so what the hell? Maybe our server logs will provide some insight.

[Thu Aug 19 13:30:50 2010] [error] [client 10.1.1.2] user \\username not found: /path/file.ps1 

 Ah ha, so our user isn't found. Thats weird though why is there a '\\' preceding our username, we didn't enter that! Lets inspect our object $creds and see if it supports this outrageous server log claim (hopefully identify the source of the hiccup).

PS C:\Users\username>$creds 

UserName              Password

--------              --------

\username             System.Security.SecureString

PS C:\Users\username>

Huh so, there is our issue. Our LDAP server certainly has no idea who '\username' is. So our Username property in our... wait what type of object is this? Running $creds.GetType() shows us its a 'PSCredential' type. The MSDN shows us this is in the 'System.Management.Automation' namespace. So in theory I can use the New-Object cmdlet to make a new object with the contents of the old object and clean it up, right? Right!

PS C:\Users\username> $creds = New-Object -TypeName 'System.Management.Automation.PSCredential' -ArgumentList $creds.Username.Remove(0,1),$creds.Password

Alright,  lets check out our object now...

PS C:\Users\username>$creds 

UserName              Password

--------              --------

username             System.Security.SecureString

PS C:\Users\username>

Very cool, so lets try it out (cross our fingers):

PS C:\Users\username> Start-BitsTransfer -Credential $creds -Source "https://ourmixbreedserver.domain.com/path/file.ps1" -Destination C:\destination\ -Authentication basic

PS C:\Users\username> 

Voilà, we're now providing our user credentials successfully and downloading the file we want without error! Powershell, you have just made my Windows using experience that much better.