Skip Ribbon Commands
Skip to main content

Home

:

Home
 

 

 

Announcement:OK so all the international travel is over. As much fun as France and Ireland were last year... I'm glad to say I'm back in the USA for a while! My recent workshops on Windows Server 2012 and Vital Signs in San Antonio, Dallas, Kansas City, Memphis, Maryland and Virginia also went great, hope you were there. I'm taking a hiatus from transactional work for a while (which sadly includes teaching workshops) and will be picking up a dedicated engagement in the Public Sector space. I feel I need to understand that vertical better (and it gets me off airplanes every week for a while). I'll move back to transactional again one day though so keep an eye out for more workshops!!

This is the BLOG and technical resource site for Chris Davis. I work for Microsoft as a Windows Platform Engineer in the consulting services division (MCS).

I'm also a magnet for some pretty odd technical issues and I love to share. You'll find that a large part of this site is dedicated to performance troubleshooting/tuning and also to a large degree, Active Directory.

This site is also a mirror for my Technet BLOG site and a launching site to my YouTube channel. So if anything here interests you, follow me on:
Join me on LinkedIn! Follow Me on Twitter! Join Facebook! YouTube!    

February 21
SQL Event ID 18456: Login failed for user Reason: Token-based server access validation failed
If you get event ID 18456 with Source MSSQLSERVER in your application and SQL logs with the verbiage of "Login failed for user 'domain\service.account.you.use'. Reason: Token-based server access validation failed with an infrastructure error. Check for previous errors" somebody may have messed with your "CONNECT SQL" permissions.
Everybody who reads this blog knows that I'm not a SQL guy, but a lot of advice on the internet suggests that this is UAC related so some of you Platform and AD folks might get asked about the error. Especially considering that the error suggests an issue with tokens and authentication, rather than permissions.
Though this isn't the only potential root cause, a deny or revoke or lack of grant can cause this. Especially on 2008 R2. In 2012 a lot of rules changed in regards to grouping but in 08R2 the loss of this would potentially cause various outages.
In our case what was happening was the configuration server was no longer accessible by the various SQL servers and this was appearing on that server every minute.
Try this:
select * from sys.server_permissions
and
select * from sys.server_principals
This may turn up that builtin\administrators or the service account (or some other important security principle) has a specific "deny" on "connect sql" or... revoke, or even a simple lack of grant.
February 20
Event ID 1057 - The Terminal Server has failed to create a new self signed certificate

If you receive Event ID 1057 - "The Terminal Server has failed to create a new self signed certificate to be used for Terminal Server authentication on SSL connections. The relevant status code was Key not valid for use in specified state" from source TerminalServices-RemoteConnectionManager in the System event log, you may have an issue with a lot of strange advice. For me, none of which worked. I finally figured out the problem.
The conditions you'll probably also notice is that you can't remote desktop into the server until you remove the "Allow connection only from computers running Remote Desktop with Network Level Authentication" checkbox in the Remote Desktop Session Host Configuration's RDP-Tcp properties General Tab or from the System settings under the Remote tab by changing the radio button back to "Allow connections from computers running any version of Remote Desktop (less secure)".
In my case I had already tried a lot of the advice like deleting the self-signed certificate and rebooting (MMC/Certificates/Local Computer/Remote Desktop) And deleting these keys and restarting:
“HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\RCM”  > Certificate “HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\RCM” > CertificateOld “HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations” > SelfSignedCertificate I also deleted the Host Configuration's RDP-Tcp connection object all together and restarted the Remote Desktop Services service.
What did finally work, I noticed that we had a bunch of crypto keys that looked like this:
C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_XXXXXXXX
I moved them all to a subfolder so there were none left in the MachineKeys folder. I then opened the MachineKeys and re-applied the full-control permission to the local server administrators group. (Security/Advanced/Change Permissions/Replace all child object permissions) and applied this.
I then restarted the Remote Desktop Services service and this time I didn't get the error about the certificate. I changed the security setting for RDP back to secure and was able to log on through Remote Desktop.

February 14
​Mitigating the Risk of Lateral Movement
In the news recently, you’ve probably read a lot about Pass the Hash.
I wanted to talk about another similar security threat that is often outlined in these whitepapers, and what we did at my customer to solve the problem.
In just about every large organization, servers and workstations are built from a standard image to save time. The problem is, those "standard" images come with a "standard" password for the local Administrator account. Oh I'm certain each of you reading this protect your Domain Admin accounts, right? Why? Because they have (in most environments) administrative rights to every computer on your network.
Guess what else has admin access to every computer on your network if you use a standardized password for your local Administrator accounts? You guessed it. All it takes is that one password which is probably the oldest one in your environment. A password which everyone who deploys servers or troubleshoots them when they're offline probably has access to. From one machine a malicious user with that information can jump laterally to every other server in your network.
There are three things all of us should do to eliminate this risk. Two of which you're probably already familiar with and perhaps are already doing. First, rename the built-in local administrator account. Second, restrict these accounts from network logons and RDP access (just use your DRAC, iLO, or crash cart). But the third is more complex, you need to make sure every single machine on your network has a unique and non-stale password for the local Administrator account.
So that's a cinch, right? Just log one-by-one into every computer in your environment. Make up some random password. Write it down on your super, um, secret-and-undoubtedly-secure Excel spreadsheet and hope nobody ever loses or misplaces that document. Riiiight. Well, what if you already had a tool that was secure, standardized, scalable,highly available, distributed, and with the capability to store stuff just like this? Well, I know one and it is called Active Directory.
A solution that I recently ran across was developed by one of our own MCS family members, and it elegantly solves this issue. It is called AdmPwd. It automatically scrambles and stores all your local Administrator passwords and stores them in AD in case you need them later. It then manages the passwords for you, changing them every 30 days (configurable). It is easy (for the right person) to find the passwords when needed and lets you force the computers to change their passwords should that become necessary. For example a local hardware admin out in a remote branch is working a server outage and needs login access. You can look up and give them that password and then easily flip a "switch" that will force the computer to change its password again the next time it is back on the network.
AdmPwd is written by MCS engineer Jiri Formacek. It is an open source GPO/CSE/PowerShell solution that is very simple to deploy, but nonetheless a very powerful tool with very few moving parts. I had this deployed in my lab in under an hour, and into production at my customer in a day.
The idea is ingenious in its simplicity. Every account on your domain exists in Active Directory as a computer security principle. And every computer in your domain has a local Administrator account, right? So that's the premise of this idea - to hook these two up.
At a high level all you need to do to deploy this solution is:
·         Download and read the AdmPwd manual from code.msdn.microsoft.com
·         Download and install the application on a management server (very small footprint)
·         Run a simple PowerShell script that automatically extends your schema *(1)
·         Create a GPO for this solution, attach it to the OU where your computers are, and run a simple PowerShell script to register that GPO with AdmPwd *(2)
·         Edit the GPO with your preferences (password complexity, etc)
·         Run a simple PowerShell script that gives users who need access to view/reset the password the appropriate rights
·         Deploy the client side extension (CSE) to your servers and workstations *(3)
 
*(1) The two schema attributes which are added are confidential, meaning they are restricted from just any low-level admin or network user with LDP to query. You will, obviously, need to be a Schema Admin for a few minutes while you run this PowerShell script.
*(2) If you're using a Central Store for your ADMX files, don't forget to copy the template and language file over.
*(3) There are a lot of ways to do this. In reality it is a simple DLL file that needs registered. You could either do this via script, GPO scheduled task (if you need it fast), or use the same GPO your registered to silently deploy the included installer on the next reboot of the computer. I recommend the latter as any new computers added to your network will pick this up going forward.
 
Some extra geek'y info FAQ. Feel free to read on if you're the one who will hands-on implement this:
 
Q: Does it encrypt the passwords in AD or is it stored in the evil clear text?
A: It is "evil" clear text. But it is stored in an attribute with the confidential searchFlags attribute set (specifically 0x288).
 
Q: OMG WHY?!?!? Why is this thing in evil clear text in a directory database? (where's my tinfoil?)
A: Because if your standard users can query confidential-marked attributes in your AD environment then you have a much bigger problem than lateral movement attacks. Actually I make fun but encrypted passwords will eventually be added to this solution to protect against offline attacks on the database. It is low priority however because again, you have MUCH bigger problems at this point.
 
Q: We renamed the local Administrator account on all our computers via GPO (or manually). Will this solution still work?
A: Yes, it uses then well-known GUID, not the display name of the account.
 
Q: We have fourteen local admin accounts on all of our servers. Will this work for all of them?
A: OMG why do you have fourteen local admin accounts? Your servers don't store any of my personal information on them do they? No, this is a solution for the local Administrator account's password. You might check into Managed Service Accounts and Group Managed Service Accounts.
 
Q: We keep our computer accounts in the Computers container, will this still work?
A: Yes, but you can't attach the GPO to the Computers container because… well, it's a container. Not an OU. So you'll need to link the GPO to the domain and that should work just fine.
 
Q: Will this work on Domain Controllers?
A: Sort of. Domain Controllers don't have local Administrator accounts, they have the Directory Service Restore Mode account. So it will "appear" to work, in that AdmPwd will populate a value. But you can't laterally attack anything with a DSRM account so it is MOOT.
 
Q: Why create a new GPO rather than leveraging an existing one?
A: This didn't work for me and I don't know why yet. The developer and I are working on it. But it worked great as soon as I created a new GPO so I'd start there.
 
Q: It isn't working. Where do I go to see if it is trying to register?
A: The Application Log on the client. If there aren’t events under source: AdmPwd saying things like "unable to register password, check permissions" it is possible the CSE is sitting idle. Change this registry key to beef up logging: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\GPExtensions\{D76B9641-3288-4f75-942D-087DE603E3EA}\ExtensionDebugLevel=2  With this on you can see it running through its execution path about every five minutes. If it isn’t, your GPO likely isn’t registered. Also run a gpresult /scope computer /H PolicyResults.htm and then view that document to be sure you’re picking up the policies you set.
 
January 23
the volume that contains the sql server data directory <volume> does not belong to the cluster group

​When you install a failover cluster, then try and add SQL to one of the nodes, you might be told on the selection screen that the volume that contains the sql server data directory <volume> does not belong to the cluster group. This had me scratching my head for a few minutes.
Turns out that you need shared storage, but don't add it to the cluster yet.
In my case I had three drives. A witness, a CSV, and a drive sitting in available storage. None of them would work. So I removed the available storage from the cluster and pointed the install at that drive and it worked. It took a few minutes while the wizard validated the storage selection but it did eventually work.
EDIT: I did this again and put the disk in maintenance mode and that also worked.

August 13
Let's Tech: Hyper-V Replica on 2012 R2 Beta

http://youtu.be/UpLFAEYLs3E

In this episode of Let's Tech I cover some of the new features of Hyper-V Replica that changed when 2012 R2 went into Beta. This is an update to my previous Replica video (On DR) that I did in the What's New In Windows Server 2012 series. Enjoy!

July 16
Let's Tech: Migrate DNS from Standard to AD Integrated

http://youtu.be/d590UT7Pvqo

In this episode I'll show you how to take a Standard Primary-Secondary DNS Zone sitting on a non-DC and migrate it easily to a Domain Controller so you can make it AD Integrated.

July 11
Let's Tech: Backing Up Windows 8

http://youtu.be/1B5_J31yycU

In today's episode we'll cover (very quickly) how to run a Windows 8 backup the way you used to run backups in Windows 7.

July 03
Let's Tech: AD LDS

http://youtu.be/i6tR6t20HO8

In today's -Let's Tech- I set up an LDS (Lightweight Directory Services, formerly ADAM or Active Directory Application Mode) instance and then do an import from AD into it. This was my first AD LDS in a long time and so it didn't work at first and so we get to see some troubleshooting (sort of the point of the Let's Tech series). I also cover some background on loose replication consistency, partitions, and why people might need an LDS instance to accomplish certain goals.

July 01
Let's Tech: Server 2012 R2 Preview

http://youtu.be/V-vLEn9gXBI

If you haven't had a chance to pull down and look at the new Server 2012 Preview that released yesterday, well... here is quick look at it. I don't go into any of the new Hyper-V stuff, just a 100-level glance at the new UI (and UX) changes and some other general roles-features and.. well... kittens, naturally. Enjoy!

June 12
What's New in Windows Server 2012, Episode 20: RDS - Remote Desktop Services & RemoteApp

http://youtu.be/nYLx4P3vL80

In this episode I will show you how to set up a simple RemoteApp and Remote Desktop Services (RDS) infrastructure in your test lab using both a self-signed cert and a real cert. How to create so published apps, etc. This is a fun one.

1 - 10Next
 

 All Articles

 
  
  
  
Category
  
  
Edit
  
No presence informationChris Davis2/21/2014 11:17 AMTechnical0 
  
No presence informationChris Davis2/20/2014 12:08 PMTechnical0 
  
No presence informationChris Davis2/14/2014 11:41 AMTechnical0 
  
No presence informationChris Davis1/23/2014 11:40 PMTechnical0 
  
No presence informationChris Davis8/13/2013 3:36 PM0 
  
No presence informationChris Davis7/16/2013 12:54 PM0 
  
No presence informationChris Davis7/11/2013 9:59 AM0 
  
No presence informationChris Davis7/3/2013 1:04 PM0 
  
No presence informationChris Davis7/1/2013 2:50 PM0 
  
No presence informationChris Davis6/12/2013 8:26 AM0 
  
No presence informationChris Davis5/20/2013 9:32 AM0 
  
No presence informationChris Davis5/17/2013 11:25 AM0 
  
No presence informationChris Davis5/15/2013 1:13 PM0 
  
No presence informationChris Davis5/14/2013 11:16 PM0 
  
No presence informationChris Davis4/5/2013 2:42 PM0 
  
No presence informationChris Davis4/4/2013 1:44 PM0 
  
No presence informationChris Davis4/3/2013 9:50 PM0 
  
No presence informationChris Davis4/2/2013 5:28 PM0 
  
No presence informationChris Davis4/1/2013 8:11 PM0 
  
No presence informationChris Davis3/29/2013 5:59 PM0 
  
No presence informationChris Davis3/28/2013 9:37 PM0 
  
No presence informationChris Davis3/27/2013 9:47 PM0 
  
No presence informationChris Davis3/16/2013 8:26 PM0 
  
No presence informationChris Davis3/15/2013 2:52 PM0 
  
No presence informationChris Davis3/14/2013 1:14 PM0 
  
No presence informationChris Davis3/13/2013 12:49 PM0 
  
No presence informationChris Davis3/12/2013 9:10 PM0 
  
No presence informationChris Davis3/12/2013 9:22 AM0 
  
No presence informationChris Davis3/11/2013 7:00 PM0 
  
No presence informationChris Davis3/11/2013 12:52 AM0 
1 - 30Next
 

 About the 9z

 
 me.jpg
Chris Davis: I am married and have two daughters. I live in Oklahoma and I work for Microsoft. I love my job because I get to work directly with customers helping them to better understand the technology they implement, perform risk assessments on their environments, and help fix things if they break. For me, this is the perfect mix... Catch me at my next workshop, hope to see you there!