Skip Ribbon Commands
Skip to main content

Home

:

Home
 

 

 

Announcement:OK so all the international travel is over. As much fun as France and Ireland were last year... I'm glad to say I'm back in the USA for a while! My recent workshops on Windows Server 2012 and Vital Signs in San Antonio, Dallas, Kansas City, Memphis, Maryland and Virginia also went great, hope you were there. I'm taking a hiatus from transactional work for a while (which sadly includes teaching workshops) and will be picking up a dedicated engagement in the Public Sector space. I feel I need to understand that vertical better (and it gets me off airplanes every week for a while). I'll move back to transactional again one day though so keep an eye out for more workshops!!

This is the BLOG and technical resource site for Chris Davis. I work for Microsoft as a Windows Platform Engineer in the consulting services division (MCS).

I'm also a magnet for some pretty odd technical issues and I love to share. You'll find that a large part of this site is dedicated to performance troubleshooting/tuning and also to a large degree, Active Directory.

This site is also a mirror for my Technet BLOG site and a launching site to my YouTube channel. So if anything here interests you, follow me on:
Join me on LinkedIn! Follow Me on Twitter! Join Facebook! YouTube!    

September 19
Performance Series: Why Is My DC Slow?

I’m starting a new grouping of videos called the “Performance Series” which will be a break-out of the “Let’s Tech” videos. I’m releasing the first two videos in this series today, and I’m really excited about them. Hope you enjoy!

In this two part Performance Series on slow Domain Controllers I cover how to identify causes of slow DC's, how to spot and fix bad LDAP queries and searches against the directory, run diagnostics on Active Directory, what tools to use and when. Then I cover how to fix various problems by optimizing queries, indexing attributes, and when different solutions are more appropriate.

Part 1:
http://youtu.be/28pWAxR1zv0
Part 2:
http://youtu.be/s1aHH2E733I

September 15
Let's Tech: Import Lots of Users into AD For Testing

V-Log Post Available Here:

http://youtu.be/cj1OsUqJaTM

Welcome to another Let's Tech... I don't particularly like having user1, user2, user3, user4 as a bed of test users. So in this episode I'll show you a more interesting way to populate your pre-prod environment. This I am doing ahead of another Let's Tech episode that I am working on about AD Performance, LDAP query troubleshooting, and how to brow beat lazy programmers. Enjoy!

July 10
Active Directory ACL’s Randomly Revert

I ran into a strange one recently. I have been trying to delegate some permissions to people to manage objects in AD. For *some* but not all, after I’d make the change, the ACL’s were reverting back. Sometimes in a few minutes and sometimes nearly an hour later. I thought I had a replication issue but… no other issues were present and nothing else was reverting back. I was pounding my head against the wall (as usual, because it is so helpful) and there was an alarm ringing in the back of my head, “Chris, you’ve heard of this before” but it finally just clicked yesterday.

Turns out that a whole handful of users in my org belonged to the “Server Operators” built-in security group. This is a protected group, as are:

  • Account Operators
  • Administrator
  • Administrators
  • Backup Operators
  • Domain Admins
  • Domain Controllers
  • Enterprise Admins
  • Krbtgt
  • Print Operators
  • Read-only Domain
  • Controllers
  • Replicator
  • Schema Admins
  • Server Operators

I wasn’t aware that Server Operators was now protected (initially they weren’t), but I guess that happened in 2000 SP4. Oops, missed that on an MCSE test exam question somewhere along the line I’m sure.

Anyway, here’s what is happening. A protected group gets its ACL’s reset back automatically (if they’ve changed) once per hour. The template for this reset is called AdminSDHolder and is located at CN=AdminSDHolder,CN=System,DC=mydomain,DC=com.

So if, for instance, you change the ACL’s on somebody’s user account and give some other group the ability to reset the password for that object… one hour later that permission will disappear and the group won’t have access to reset the password. But only if that user account you tried to change is a member of one of the protected groups above.

The PDC Emulator runs this job once every 60 minutes. So after making the change, you won’t even notice that the problem for up to an hour. If fact, you’ll think you finished the task.

To make things worse, the box for inheriting permissions will uncheck as well – meaning if you tried to delegate a whole OU it will work for some accounts and not for others.

So… how to fix it? For me I just yanked them out of the Server Operators group and that fixed it. But if you need to keep the security group mentioned above, you have two options. Modify AdminSDHolder with new permissions or take the group in question out from under the protection of AdminSDHolder. Ned (as usual) has a good article about these options here so no need to go into them and re-invent the wheel.

February 21
SQL Event ID 18456: Login failed for user Reason: Token-based server access validation failed
If you get event ID 18456 with Source MSSQLSERVER in your application and SQL logs with the verbiage of "Login failed for user 'domain\service.account.you.use'. Reason: Token-based server access validation failed with an infrastructure error. Check for previous errors" somebody may have messed with your "CONNECT SQL" permissions.
Everybody who reads this blog knows that I'm not a SQL guy, but a lot of advice on the internet suggests that this is UAC related so some of you Platform and AD folks might get asked about the error. Especially considering that the error suggests an issue with tokens and authentication, rather than permissions.
Though this isn't the only potential root cause, a deny or revoke or lack of grant can cause this. Especially on 2008 R2. In 2012 a lot of rules changed in regards to grouping but in 08R2 the loss of this would potentially cause various outages.
In our case what was happening was the configuration server was no longer accessible by the various SQL servers and this was appearing on that server every minute.
Try this:
select * from sys.server_permissions
and
select * from sys.server_principals
This may turn up that builtin\administrators or the service account (or some other important security principle) has a specific "deny" on "connect sql" or... revoke, or even a simple lack of grant.
February 20
Event ID 1057 - The Terminal Server has failed to create a new self signed certificate

If you receive Event ID 1057 - "The Terminal Server has failed to create a new self signed certificate to be used for Terminal Server authentication on SSL connections. The relevant status code was Key not valid for use in specified state" from source TerminalServices-RemoteConnectionManager in the System event log, you may have an issue with a lot of strange advice. For me, none of which worked. I finally figured out the problem.
The conditions you'll probably also notice is that you can't remote desktop into the server until you remove the "Allow connection only from computers running Remote Desktop with Network Level Authentication" checkbox in the Remote Desktop Session Host Configuration's RDP-Tcp properties General Tab or from the System settings under the Remote tab by changing the radio button back to "Allow connections from computers running any version of Remote Desktop (less secure)".
In my case I had already tried a lot of the advice like deleting the self-signed certificate and rebooting (MMC/Certificates/Local Computer/Remote Desktop) And deleting these keys and restarting:
“HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\RCM”  > Certificate “HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\RCM” > CertificateOld “HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations” > SelfSignedCertificate I also deleted the Host Configuration's RDP-Tcp connection object all together and restarted the Remote Desktop Services service.
What did finally work, I noticed that we had a bunch of crypto keys that looked like this:
C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\f686aace6942fb7f7ceb231212eef4a4_XXXXXXXX
I moved them all to a subfolder so there were none left in the MachineKeys folder. I then opened the MachineKeys and re-applied the full-control permission to the local server administrators group. (Security/Advanced/Change Permissions/Replace all child object permissions) and applied this.
I then restarted the Remote Desktop Services service and this time I didn't get the error about the certificate. I changed the security setting for RDP back to secure and was able to log on through Remote Desktop.

February 14
​Mitigating the Risk of Lateral Movement
In the news recently, you’ve probably read a lot about Pass the Hash.
I wanted to talk about another similar security threat that is often outlined in these whitepapers, and what we did at my customer to solve the problem.
In just about every large organization, servers and workstations are built from a standard image to save time. The problem is, those "standard" images come with a "standard" password for the local Administrator account. Oh I'm certain each of you reading this protect your Domain Admin accounts, right? Why? Because they have (in most environments) administrative rights to every computer on your network.
Guess what else has admin access to every computer on your network if you use a standardized password for your local Administrator accounts? You guessed it. All it takes is that one password which is probably the oldest one in your environment. A password which everyone who deploys servers or troubleshoots them when they're offline probably has access to. From one machine a malicious user with that information can jump laterally to every other server in your network.
There are three things all of us should do to eliminate this risk. Two of which you're probably already familiar with and perhaps are already doing. First, rename the built-in local administrator account. Second, restrict these accounts from network logons and RDP access (just use your DRAC, iLO, or crash cart). But the third is more complex, you need to make sure every single machine on your network has a unique and non-stale password for the local Administrator account.
So that's a cinch, right? Just log one-by-one into every computer in your environment. Make up some random password. Write it down on your super, um, secret-and-undoubtedly-secure Excel spreadsheet and hope nobody ever loses or misplaces that document. Riiiight. Well, what if you already had a tool that was secure, standardized, scalable,highly available, distributed, and with the capability to store stuff just like this? Well, I know one and it is called Active Directory.
A solution that I recently ran across was developed by one of our own MCS family members, and it elegantly solves this issue. It is called AdmPwd. It automatically scrambles and stores all your local Administrator passwords and stores them in AD in case you need them later. It then manages the passwords for you, changing them every 30 days (configurable). It is easy (for the right person) to find the passwords when needed and lets you force the computers to change their passwords should that become necessary. For example a local hardware admin out in a remote branch is working a server outage and needs login access. You can look up and give them that password and then easily flip a "switch" that will force the computer to change its password again the next time it is back on the network.
AdmPwd is written by MCS engineer Jiri Formacek. It is an open source GPO/CSE/PowerShell solution that is very simple to deploy, but nonetheless a very powerful tool with very few moving parts. I had this deployed in my lab in under an hour, and into production at my customer in a day.
The idea is ingenious in its simplicity. Every account on your domain exists in Active Directory as a computer security principle. And every computer in your domain has a local Administrator account, right? So that's the premise of this idea - to hook these two up.
At a high level all you need to do to deploy this solution is:
·         Download and read the AdmPwd manual from code.msdn.microsoft.com
·         Download and install the application on a management server (very small footprint)
·         Run a simple PowerShell script that automatically extends your schema *(1)
·         Create a GPO for this solution, attach it to the OU where your computers are, and run a simple PowerShell script to register that GPO with AdmPwd *(2)
·         Edit the GPO with your preferences (password complexity, etc)
·         Run a simple PowerShell script that gives users who need access to view/reset the password the appropriate rights
·         Deploy the client side extension (CSE) to your servers and workstations *(3)
 
*(1) The two schema attributes which are added are confidential, meaning they are restricted from just any low-level admin or network user with LDP to query. You will, obviously, need to be a Schema Admin for a few minutes while you run this PowerShell script.
*(2) If you're using a Central Store for your ADMX files, don't forget to copy the template and language file over.
*(3) There are a lot of ways to do this. In reality it is a simple DLL file that needs registered. You could either do this via script, GPO scheduled task (if you need it fast), or use the same GPO your registered to silently deploy the included installer on the next reboot of the computer. I recommend the latter as any new computers added to your network will pick this up going forward.
 
Some extra geek'y info FAQ. Feel free to read on if you're the one who will hands-on implement this:
 
Q: Does it encrypt the passwords in AD or is it stored in the evil clear text?
A: It is "evil" clear text. But it is stored in an attribute with the confidential searchFlags attribute set (specifically 0x288).
 
Q: OMG WHY?!?!? Why is this thing in evil clear text in a directory database? (where's my tinfoil?)
A: Because if your standard users can query confidential-marked attributes in your AD environment then you have a much bigger problem than lateral movement attacks. Actually I make fun but encrypted passwords will eventually be added to this solution to protect against offline attacks on the database. It is low priority however because again, you have MUCH bigger problems at this point.
 
Q: We renamed the local Administrator account on all our computers via GPO (or manually). Will this solution still work?
A: Yes, it uses then well-known GUID, not the display name of the account.
 
Q: We have fourteen local admin accounts on all of our servers. Will this work for all of them?
A: OMG why do you have fourteen local admin accounts? Your servers don't store any of my personal information on them do they? No, this is a solution for the local Administrator account's password. You might check into Managed Service Accounts and Group Managed Service Accounts.
 
Q: We keep our computer accounts in the Computers container, will this still work?
A: Yes, but you can't attach the GPO to the Computers container because… well, it's a container. Not an OU. So you'll need to link the GPO to the domain and that should work just fine.
 
Q: Will this work on Domain Controllers?
A: Sort of. Domain Controllers don't have local Administrator accounts, they have the Directory Service Restore Mode account. So it will "appear" to work, in that AdmPwd will populate a value. But you can't laterally attack anything with a DSRM account so it is MOOT.
 
Q: Why create a new GPO rather than leveraging an existing one?
A: This didn't work for me and I don't know why yet. The developer and I are working on it. But it worked great as soon as I created a new GPO so I'd start there.
 
Q: It isn't working. Where do I go to see if it is trying to register?
A: The Application Log on the client. If there aren’t events under source: AdmPwd saying things like "unable to register password, check permissions" it is possible the CSE is sitting idle. Change this registry key to beef up logging: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\GPExtensions\{D76B9641-3288-4f75-942D-087DE603E3EA}\ExtensionDebugLevel=2  With this on you can see it running through its execution path about every five minutes. If it isn’t, your GPO likely isn’t registered. Also run a gpresult /scope computer /H PolicyResults.htm and then view that document to be sure you’re picking up the policies you set.
 
January 23
the volume that contains the sql server data directory <volume> does not belong to the cluster group

​When you install a failover cluster, then try and add SQL to one of the nodes, you might be told on the selection screen that the volume that contains the sql server data directory <volume> does not belong to the cluster group. This had me scratching my head for a few minutes.
Turns out that you need shared storage, but don't add it to the cluster yet.
In my case I had three drives. A witness, a CSV, and a drive sitting in available storage. None of them would work. So I removed the available storage from the cluster and pointed the install at that drive and it worked. It took a few minutes while the wizard validated the storage selection but it did eventually work.
EDIT: I did this again and put the disk in maintenance mode and that also worked.

August 13
Let's Tech: Hyper-V Replica on 2012 R2 Beta

http://youtu.be/UpLFAEYLs3E

In this episode of Let's Tech I cover some of the new features of Hyper-V Replica that changed when 2012 R2 went into Beta. This is an update to my previous Replica video (On DR) that I did in the What's New In Windows Server 2012 series. Enjoy!

July 16
Let's Tech: Migrate DNS from Standard to AD Integrated

http://youtu.be/d590UT7Pvqo

In this episode I'll show you how to take a Standard Primary-Secondary DNS Zone sitting on a non-DC and migrate it easily to a Domain Controller so you can make it AD Integrated.

July 11
Let's Tech: Backing Up Windows 8

http://youtu.be/1B5_J31yycU

In today's episode we'll cover (very quickly) how to run a Windows 8 backup the way you used to run backups in Windows 7.

1 - 10Next
 

 All Articles

 
  
  
  
Category
  
  
Edit
  
No presence informationChris Davis9/19/2014 8:56 AM0 
  
No presence informationChris Davis9/15/2014 4:07 PM0 
  
No presence informationChris Davis7/10/2014 1:12 PM0 
  
No presence informationChris Davis2/21/2014 11:17 AMTechnical0 
  
No presence informationChris Davis2/20/2014 12:08 PMTechnical0 
  
No presence informationChris Davis2/14/2014 11:41 AMTechnical0 
  
No presence informationChris Davis1/23/2014 11:40 PMTechnical0 
  
No presence informationChris Davis8/13/2013 3:36 PM0 
  
No presence informationChris Davis7/16/2013 12:54 PM0 
  
No presence informationChris Davis7/11/2013 9:59 AM0 
  
No presence informationChris Davis7/3/2013 1:04 PM0 
  
No presence informationChris Davis7/1/2013 2:50 PM0 
  
No presence informationChris Davis6/12/2013 8:26 AM0 
  
No presence informationChris Davis5/20/2013 9:32 AM0 
  
No presence informationChris Davis5/17/2013 11:25 AM0 
  
No presence informationChris Davis5/15/2013 1:13 PM0 
  
No presence informationChris Davis5/14/2013 11:16 PM0 
  
No presence informationChris Davis4/5/2013 2:42 PM0 
  
No presence informationChris Davis4/4/2013 1:44 PM0 
  
No presence informationChris Davis4/3/2013 9:50 PM0 
  
No presence informationChris Davis4/2/2013 5:28 PM0 
  
No presence informationChris Davis4/1/2013 8:11 PM0 
  
No presence informationChris Davis3/29/2013 5:59 PM0 
  
No presence informationChris Davis3/28/2013 9:37 PM0 
  
No presence informationChris Davis3/27/2013 9:47 PM0 
  
No presence informationChris Davis3/16/2013 8:26 PM0 
  
No presence informationChris Davis3/15/2013 2:52 PM0 
  
No presence informationChris Davis3/14/2013 1:14 PM0 
  
No presence informationChris Davis3/13/2013 12:49 PM0 
  
No presence informationChris Davis3/12/2013 9:10 PM0 
1 - 30Next
 

 About the 9z

 
 me.jpg
Chris Davis: I am married and have two daughters. I live in Oklahoma and I work for Microsoft. I love my job because I get to work directly with customers helping them to better understand the technology they implement, perform risk assessments on their environments, and help fix things if they break. For me, this is the perfect mix... Catch me at my next workshop, hope to see you there!