Scripting to solve problems (Windows DNS Debug Logs, Splunk, Powershell)

Anyone that’s worked in or been involved in IT for any substantial amount of time knows that creating scripts, even the most simple ones, can solve a lot of your problems.

A while back, I started pulling DNS Requests Debug Logs from my Windows DNS servers in to Splunk.  Having a copy of request logs, and being able to drill down into them with Splunk, can be extremely helpful for many reasons, including if your IPS alerts you to some suspicious DNS queries and you need to find the origin.

Unfortunately, Splunk occasionally causes one of my DNS servers to be unable to write to the debug log, and the server writes Event ID 3152 to the application log.  The details that are logged say:

The DNS server was unable to open file D:\DNS Requests Log\dnsrequests.txt for write.  Most likely the file is a zone file that is already open.  Close the zone file and re-initiate zone write.

I don’t know what exactly is causing the failure, but some other people have also noticed the problem.

To work around this problem, I wrote a short Powershell script and triggered it to run (via Task Scheduler) whenever Event ID 3152 is logged.  This works well for my environment because I’m not seeing Event ID 3152 logged for any other reasons than when the DNS Server service is unable to write to the debug log.  It can also be logged due to problems writing to a zone file (https://technet.microsoft.com/en-us/library/cc735838(v=ws.10).aspx) so you should research your own environment before implementing this.  Actually, you should definitely, absolutely, always, research your own environment before you believe what some random guy on the Internet wrote.

I started out with a 1 line script to just restart the service, but on one occasion, the script triggered, and immediately logged 3152 again (probably while the originally triggered instance of the task was still running), and the DNS Server service was never able to create the new log file to write to.  So, I ended up with a script that stops the SplunkForwarder service, restarts the DNS Service, sleeps for 5 seconds, and then checks to make sure that the file exists and restarts the service again if it still doesn’t exist, and then starts SplunkForwarder.

$DNSLogPath = “D:\DNS Requests Log\dnsrequests.txt”

Stop-Service SplunkForwarder

Restart-Service DNS

Start-Sleep 5

if(!(Test-Path -Path $DNSLogPath))
{
Start-Service SplunkForwarder
}
else
{
Restart-Service DNS

Start-Sleep 5

Start-Service SplunkForwarder
}

exit

Of all things, my scripting/programming experience is the weakest, so get in touch with me if you think this is ugly and have a better way to do it!

Also, once you’ve got the script scheduled to run based on an Event Viewer trigger, you can add a second action to the Task Scheduler job that will send you an email.  That way, you know when it happens and can manually verify that the new debug log exists.

Scripting to solve problems (Windows DNS Debug Logs, Splunk, Powershell)