I've nearly given up blogging as you can probably tell by the gaps between my posts these days and am uncertain whether I'll ever get that going again or not.  Who knows?  However, something weird happened today related to ADFS that I thought might be worth a post that I decided to document for posterity.

Essentially, a colleague of mine at work was trying to install the ADFS Web Agent on a Win2K3 R2 server using the standard method of "Add/Remove Windows Components".  The OS was running on a VM with the image built through some mechanism that neither of us had any firsthand knowledge of.  Essentially, when he tried to run the install, he got an error saying basically that the install failed.  In the event log was something similar pointing to a log file that showed the command line used to run MSIEXEC which in turn included a command line option to provide additional logging in another file.  That file was very short by MSI logging standards and basically recorded the error "1620" which translates to "not a valid intaller package":

C:\>err 1620
# for decimal 1620 / hex 0x654 :
  ERROR_INSTALL_PACKAGE_INVALID                                 winerror.h
# This installation package could not be opened.  Contact the
# application vendor to verify that this is a valid Windows
# Installer package.
# 1 matches found for "1620"

This is weird!  The file c:\windows\adfs.msi is not a valid installer package?

So, I grabbed the file and copied it to a machine that has Orca installed.  Orca is a tool used by MSI developers to examine the contents of MSI databases (you can get it for free with the Platform SDK tools).  When I opened the database, it was clear that this was not actually an MSI database at all but instead an MSI patch database (usually a .MSP file).  Strange.  When I went back to the server and looked, there was also a file called ADFS.MSP in C:\windows with the same size.  They turned out to be the same file with different names!  So, something had dorked up the file images associated with the ADFS component.

• To solve the problem, I first did a file search for adfs.msi and found a copy in an R2 source directory locally
• I then copied this file into the c:\windows directory (after renaming the other file "just in case")
• Then, upon running Add/Remove Windows Components, the installer started to work correctly.  I WAS reprompted to provide the location of "adfs.msi" and put in the file path to the R2 source where I found it.  After that, all was well.  Hopefully the patch got applied correctly as well (I didn't check). 

I have no idea how the box got dorked up or if this affects anyone by my colleagues on our servers, but I thought it was worth mentioning. This is the first time I've ever had any success troubleshooting an ADFS installer issue, so it seemed worthy of a post.  :)

It seems like I've deteriorated into semi-annual blog posts.  Sigh.  At least the discussion groups at www.directoryprogramming.net continue to flourish and we are seeing a nice uptick in activity on the ADFS board there.  I think the writing may be on the wall for me as a blogger, but who knows.  Perhaps I'll get back on the wagon.

Anyway, thanks to all the people who came to see my talks at DEC this year.  I hope you enjoyed visiting my town and you got a lot out of the conference itself.  DEC is one of my favorites and I'm happy to see it continue to do well.  I got a lot of nice feedback on both of my talks and I'm always interested to hear what you thought.

My first talk this year was on customizing ADFS.  To my knowledge, this type of stuff has never been talked about publicly before, so the session was a bit of an experiment.  I essentially tried to cover all the different types of things you should and could do to ADFS V1 or 1.1 (2003 R2 server or 2008 server) to make it do different things.  It started off discussing some cosmetics to apply to the pages displayed by the FS, moved through ADAM account store tweaks and then covered custom claim transformation modules and some advanced hacks/mods such as non-Windows authentication.  Some of that stuff has been discussed here on this blog and nearly all of it was based on stuff we've actually done at work, so basically real world experience.  My fear was that the subject matter would be a little over the audience's head since it wasn't introductory at all (assumed you already knew ADFS pretty well) and covered some developer stuff which might have seemed alien to many of the audience members who tend to be more of the IT Pro sort.  Still, I think it was valuable stuff.

I mentioned a sample custom claim transform module that I'd post the source to.  I'll follow that up in a separate post shortly.

The audiences for all the ADFS talks were pretty small by comparison to the big AD talks, but this doesn't surprise me.  Not only is ADFS still a new thing, but to a great extent it is a luxury for most DEC attendees to be able to go to those sessions.  ADFS largely assumes that both the directory and the identity provisioning stuff is all a solved problem and that we have rich repositories filled with security principals whose identity is trustworthy and stuffed with useful metadata that can be converted to claims.  For many, that is not (yet) a solved problem at all and federated identity is still largely wishful thinking.  Still, you have to start somewhere.

My second talk was essentially the talk I've done at DEC now for 3 years, although this time modified heavily to cover the new .NET 3.5 stuff in System.DirectoryServices.AccountManagement.  I was really dreading this talk and didn't finish the slides for it before the conference, so between the last minute PPT work and the anxiety, I only got about an hour of sleep.

Still, the talk seemed to go off pretty well.  I was in the big room this time and in one of the last slots of the conference, so I had low expectations for turnout.  It seemed like I had a pretty good crowd though (not the Dean and Joe show, but I'll take it!) and people seemed to be into it.  I think it was probably the best version of that talk I've done to date, so all in all I'm quite happy.  I miss having Ryan there to bounce stuff around with, but so it goes.

Thanks also to Donovan for inviting me up for his case studies talk and giving me an opportunity to talk about some of the real world stuff we are doing with federation at work and talking about the process and legal stuff as much as the technical aspects.  People clearly have as many if not more questions about those things than the engineering parts.

I think I finally get CardSpace now, especially as it applies to the enterprise, and am looking forward to having a chance to get it running internally.  That should be interesting.  Stuart, I need some bits I can actually deploy.  :)  Thanks to Pamela Dingle for coming to DEC and bringing both the CardSpace love and the non-MS platform perspective.  I'm anxious to go to a conference where everyone knows her and no one knows me at all and see how I do.

She's got a nice follow up on the Wook Lee Challenge this year which I played a tiny role in.

Customizing-ADFS.zip (1.06 MB)

DotNet-DS-Programming.zip (1.27 MB)

I was lucky enough to attend DEC again this year and was even more lucky to have been asked to speak due to an unfortunate last minute cancellation.  This year, I presented on a variation of the same type of stuff that Ryan and I presented on at DEC 2006.  This year, I had to fly solo as Ryan could not attend.  :(

Here's what we did differently this time around:

• No PowerShell (DEC already had 2 PowerShell sessions, so why bother?)
• Focus on some new Longhorn LDAP and AD features (Fine Grained Password Policy)
• A "slideware" overview of what's coming in .NET 3.5 "Orcas" with the new System.DirectoryServices.AccountManagement namespace (formerly known as the Principal API).

I'd like to thank all of those who attended.  I hope you enjoyed the talk and hope that some of you got free books.  I apologize if I could not accomodate all of you.  :(  Thanks to the Addison-Wesley marketing team for providing the books for your enjoyment.

For those of you interested in the Snippet Compiler tool I used in my demos, you can find it here.

The slides and code for the demos are attached and I did get around to converting them to VB for all of you VB people (I'm a VB.NET guy too; I really don't know why I coded all the demos in C# :)). 

Note that my application of the "in chain" matching rule turns out to be incorrect usage.  Don't do it like that!  Read more here.  I feel silly.

Note that if you are confused about which API to use, S.DS or S.DS.P, I discussed that in some detail here.  There is really no right answer, but hopefully that helps. 

To ask us any specific questions about LDAP programming, please use the book's discussion forum.  This is the only place that Ryan and I both use together.

As always, DEC is a treat and I really enjoyed all the conversations and interaction and am happy to see ADFS gaining a little traction.  Now, about that hot chicken...

I've been using ADFS day in and day out for almost a year now and have run into a fairly stunning number of issues.  Most of these are simple problems with simple solutions, but they can cause hours or days of frustration to resolve.  One of the main themes that seems to pervade these little doses of IT misery is based on a misunderstanding of how cookies work and how ADFS uses cookies.  This post is my attempt to gather some of the lessons I've learned and hopefully make your time with ADFS a little easier. 

How Cookies Work in Web Browsers

Although many of us have a basic, foggy notion about what cookies do in web applications, many of us don't know all of the little rules that govern their behavior.  I'll try to run that down now based on my own understanding, which I believe is at least technically accurate enough for the purposes of the rest of this article.  Feel free to correct me if you know better or refer to the RFCs if you want more precision.

Basically, a cookie is a small piece of textual data that is sent by a web server to a web browser.  The job of the browser is to return this piece of data back to the server when it makes another request to the server, based on the rules that I'll detail shortly.  Cookies are generally used in web applications as a means of maintaining state in the web server across multiple requests made by the browser.

Cookies are implemented as well-known HTTP headers in the request data sent by the browser and the response data sent.  The request header name is called Cookie and the response header name is Set-Cookie.  The server always initiates the usage of cookies by returning a response with a Set-Cookie header in it that contains the data and some other instructions.  The browser's job is to then add a Cookie header on all subsequent requests to the server that contains exactly the data the server sent in the Set-Cookie header.  However, exactly when the browser will add the Cookie header to its request is based on the other instructions in the Set-Cookie header and the rules for cookie behavior. 

First, let's examine the "other instructions".  The Set-Cookie header may contain these bits of info:

• Name
• Value
• Domain (optional)
• Path (optional)
• Expires (optional)
• Secure flag (optional)
• HttpOnly flag (optional)

Cookie Name and Value

The Cookie Name is just what it sounds like.  Since a web app may use many different cookies, this is how they can be distinguished.  The Cookie Value is just some textual data.  It could be anything.  Oftentimes, it may contain some data that has been encrypted that the browser can't even interpret it--only the server can.  This is not a problem.  The data is for the server anyway.  The browser's job is to dutifully sending it back to the server, not understand it.

Cookie Expires, Secure and HttpOnly

I'll skip ahead to the last three, as they easy to explain and not terribly problematic (for our purposes).  The optional Expires field may contain a date/time value that tells the browser when to stop sending the cookie back to the server.  This one is interesting because it alone determines whether a cookie is a session cookie or a file-based cookie. 

If there is no expiration specified, the cookie defaults to a session cookie and will be returned as long as the browser session continues.  In IE and FireFox (and probably most other browsers on Windows), the session is basically the duration that the browser process is open.  Note that a browser process may span many windows and may originate in a non-obvious place like an email or an instant messenger client, so you may sometimes be surprised to see a cookie being sent back to a server even after a window was closed (or seemingly all windows for that matter!). 

If the expiration date is set, then the cookie is "file-based" and will be written to disk.  It will survive across multiple browser processes being launched and stopped.  The browser will continue to send it until the expiration time is exceeded.

The Secure flag instructs the browser to only return the cookie back to the server if the request uses the HTTPS (SSL) protocol.  This flag is often used when the cookie contains sensitive data that could be useful by a bad guy if it was intercepted on the wire.  I consider it to be bad practice to ever use any kind of forms-based authentication cookie without the Secure flag set, although this then requires SSL.  If you are serious about security, you use SSL for secure websites.  End of story.

The HttpOnly flag is the new guy and not all browsers support it (although many do).  It basically says that this cookie will only be sent on the wire.  The browser will not allow the cookie to show up in the browser's Document Object Model (DOM) cookie collection.  This prevents scripts from being able to view and manipulate the cookie and can help prevent some kinds of cross-site scripting (XSS) attacks.

Cookie Domain

For our purposes, The Cookie Domain and Path are the most interesting values, as this is the heart of all of the trouble in ADFS, so we cover them last.  The domain specifies which hosts the browser will replay the cookie to.  If the domain is not specified, the browser will only replay the cookie to host that sent the Set-Cookie header in the first place.  Here, the host is determined by the host portion of the URL.  For example, in http://www.joekaplan.net/default.aspx, the host is www.joekaplan.net. 

Domains can also be specified hierarchically.  I could set the cookie domain to .joekaplan.net and then my browser would return the cookie to any host within the joekaplan.net DNS namespace such as www.joekaplan.net, blogs.joekaplan.net and junk.joekaplan.net (neither of which exist in the latter two examples, but that isn't important).

One might wonder if one can set a cookie for a domain other than the domain of the host.  The answer is "it depends", but usually no.  As one can imagine, this could (and has!) lead to all kinds of crazy hacking where one site writes over another site's cookies.  In IE, the ability to do this is locked down by the various security levels and assigned by zone.  I haven't seen many legitimate uses for such a design personally, so I'm happy this is restricted now.

Another important picky detail about the domain value is that it doesn't consider the port component of the hostname in the URL.  Basically, this means that it treats the host www.joekaplan.net:80 and www.joekaplan.net:8080 as the same host, even if those map to completely different websites on your web server.  This can lead to surprises.

Cookie Path

The final modifier is the Path.  The path value is like the domain, except that is specifies the scope in the path hierarchy that the cookie will be replayed to.  Here, the path is the part of the URL that comes after the host name.  If the path is set to /, then the cookie will be replayed to any part of the hierarchy of the website.  If the path is set to /site, then the cookie will only be replayed to requests for URLs at or below /site.

One tricky detail with the path value is that it is case-sensitive.  If you set the path to /site and your URL is http://www.joekaplan.net/Site/default.aspx (note the capital "S"), guess what?  The cookie will not be returned with the request!  This little gotcha can easily confound IIS developers, as IIS is generally quite happy to treat folders and virtual directories as case-insensitive.  Developers on "other" web server platforms where URLs (and file names) are case-sensitive tend to be a little more hip to this. :)

A Brief Tour of the ADFS Cookies

ADFS, like most other HTTP authentication protocols that don't use the built in HTTP authentication specifications (Basic, Digest, Negotiate, Kerberos, NTLM, etc.), use cookies for a lot of stuff.  There are three primary cookies to know about:

• Authentication/Token cookie (_WebSsoAuth)
• Home Realm Cookie (_LSRealm)
• Logout Cookie (_LSCleanup)

Let's take each in turn.

Authentication/Token Cookie

The _WebSsoAuth cookie is certainly the most important one, as that is the cookie that represents the user's login to a federation resource.  This cookie is always a session cookie, in that the Expires field is never set, so it goes away when the browser process does.  Expiration of this cookie is handled by timestamps set in the data in the cookie itself and is a programmatic feature of the protocol and is not related to how the browser treats cookie expiration. 

The cookie basically contains the SAML 1.1 token issued by a federation server that allows access to a specific resource.  The format of the cookie for ADFS is in some crazy compressed format that MS came up with that isn't documented, but that is supposedly cool because the WS-Federation PRP spec doesn't actually contain any details about what this should look like and leaves it implementation specific.  These cookies are only consumed by the server/app that issued them (hopefully!), so this isn't a big deal.  Sometimes you may see a _WebSsoAuth0 cookie or additional numbering.  This can happen if the SAML token is very large and the compressed size of the XML in the SAML token may exceed the allowed size of a cookie.  ADFS breaks these up for you to overcome this.  This is a picky and unimportant detail for our discussion, but I figured I'd mention it in case you saw it and were curious.

The Set-Cookie header for this cookie usually looks something like this (data clipped for brevity):

Set-Cookie: _WebSsoAuth=eNrNWWuTqkgS9acYzpfd6LV5qAhGd8cWDxUVFMVnbMREASWiQ.....CdagA; path=/adfs/ls/; secure; HttpOnly

In this case, this is the authentication cookie issued by the federation server itself.  In ADFS, each federation server the user logs in to (at least one, maybe two if there is an account partner/resource partner set up) will issue a _WebSsoAuth cookie.  Additionally, each application that the user visits will issue a _WebSsoAuth cookie that is specific to the application. 

Each _WebSsoAuth cookie is different and is intended only for the app/server that issued it.  However, you probably noticed how they all have the same name.  Because they all have the same name, they must different by domain and path so that the browser will know which cookie to send to which app/server.  If you've ever configured an ADFS token-based or claims-based application, you will have noticed that you are required to specify the domain and path for the cookie when configuring the application.  All it takes is a naive mistake with this configuration to cause the browser to start sending the wrong cookie to the wrong place and ADFS chaos ensues.

Home Realm Cookie

The _LSRealm cookie is a file-based cookie issued by the resource federation server after you have successfully logged in that specifies the federation server URI (usually urn:federation:foo or something) of the server that authenticated you.  This is done so that the resource federation server doesn't have to prompt you for your home organization again if it has multiple potential partners.  We already discussed a little bit about these values and home realm discovery query strings in a post last summer, so I won't get into much more detail.  These things rarely cause trouble except that you have to clear your cookies if you want to get a resource federation server to prompt you for your home realm again after you log in the first time.  They are issued for one month as I recall; I'm too lazy to check for certain right now.  :)

Logout Cookie

The _LSCleanup cookie assists in the logout process by keeping track of which federation servers and applications you've visited.  When ADFS does a log out, it attempts to "drain the swamp" by logging you out of everything.  To do this, the built-in logout page attempts to visit the logout URL for every federation server you've visited and every app you've visited.  These logout URLs set the _WebSsoAuth cookie value for the cookies they issued to "null".  This cookie is also a session cookie.  These cookies can get screwed up just like the authentication cookies, but usually this fact never comes up because once the authentication cookies are hosed, things tend to break before you ever try to log out.

Common ADFS Problems Caused by Incorrect Cookie Settings

There are two ADFS problems I've seen frequently that cause much pain that are related to cookie problems and probably some variants that I'm forgetting:

• Invalid SAML Audience
• Infinite Loop Detected

Invalid SAML Audience

This occurs when one ADFS application get's another ADFS application's cookie.  Like I said before, each application has an application-specific cookie and there is a piece of data in the SAML token called the audience that says exactly where the cookie was supposed to go.  This is identified by the application's configured URL in the trust policy and the web site. 

There are a variety of ways you can create this error.  One easy way is by setting the cookie domain to something like .domain.com and having your apps set up with host names like app1.domain.com and app2.domain.com.  Since these are both basically children of .domain.com, the browser will happily send a _WebSsoAuth cookie to app2 that app1 issued if the paths are the same.  Since the path is often set to /, this is not uncommon.  You can also have pseudo-chaos that is even more difficult to debug if the path's aren't the same but contain some overlap in hierarchy.

I'm still trying to figure out a reason why you would ever set the cookie domain in an ADFS system.  Generally, you only ever want an ADFS cookie going back to the host that issued it, so why not leave it at the default setting of "null"?  In my experience, nothing but trouble ever comes from setting the domain.  If you think of a use case for setting this, please let me know.

I also mentioned overlapping paths.  These are a significant cause of trouble.  Let's say you have an app at the root of the site / and an app below that /Site.  Basically, if they have the same host name, you can't do that in ADFS.  They need to be strict siblings like /Site1 and /Site2. 

Overlapping paths can show up in subtle ways as well.  Sometimes, you may not think that overlapping paths are a problem because you aren't aware that you have overlapping domains.  If you use the ADFS Step-by-Step Guide to put together your ADFS demo lab, you may find yourself using host names like this to try out different application styles and using different TCP/IP ports for SSL to split them up in IIS:

• https://apps.domain.com:8081/
• https://apps.domain.com:8082/

While this makes things easy from the IIS perspective in terms of setting up multiple sites and it makes it easier to deal with few host names (and less DNS and certificate mucking around), remember that the browser treats those two hosts as being the same thing for cookie purposes.  You now have overlapping domains.  Therefore, it simply falls to the cookie path to make a distinction between them.  If you use a path like:

• /claimsapp
• /tokenapp

...for each one, you are fine, but if you naively put one or both of those apps in the root, you will have problems if you try to do SSO between them with the previously mentioned error.

Infinite Loop Errors and the Like

The other problem we can easily create is the "infinite loop" problem or "same token request within the last 20 seconds" problem on the federation server.  I've seen this happen when we have a capitalization different between the browser URL and the cookie path.  As we explained above, the cookie path is treated as case sensitive by the browser, but IIS is generally very happy to treat a URL folder or virtual directory path as case-insensitive.  The problem then manifests itself in ADFS with an interaction like this:

1. User logs into federation server after being redirected to do so by the application they were trying to access
2. Federation server generates SAML token for resource application and redirects browser to POST the token to that URL (app.domain.com/site/)
3. Resource application receives the token via POST, verifies it and issues the browser a cookie with the token contents for use with subsequent visits.  The path in the Set-Cookie header is /Site.  App redirects the user back to itself with the exact URL originally requested.
4. Browser receives the cookie, but it is using a URL like https://app.domain.com/site/default.aspx, not https://app.domain.com/Site/default.aspx.  Since those are clearly different (note the capital S!), the browser does not include the authentication cookie in the request.
5. App receives the request from the browser without the cookie and decides "this guy should log in if I'm going to allow access, so go back to my federation server, get a token and I'll see you later" and redirects the user back to the federation server with instructions to sign in.
6. Federation server is clever and notices that it just issued a token for this user for this app a second ago or so and throws an error because it knows that something just isn't right.

This one is a little subtle, but easily fixable.

Conclusion

Cookies are at the heart and soul of how ADFS works (with the Passive Requester Profile anyway), so understanding the rules that govern their behavior in the browser and how ADFS expects you to use them will save you potentially hours or days of troubleshooting misery. 

Tools for Viewing Headers and Cookies

When working with ADFS (or really just about any web app I've found), having a way to view the raw request and response headers associated with all of the traffic generated by the browser and server is invaluable.  There are great plugins for IE (ieHttpHeaders) and FireFox (Live Headers) that do a fine job of this and are free. 

(edited March 4, 2007, to add some additional formatting, fix some wording and add the section on viewing headers)

So, let's say you are doing some ADFS programming and you see this big cookie being passed to your app called _WebSsoAuth.  Let's say your app is a token app, so the only trace of the claims from the federation is in the resulting Windows token with the user and group info (see here if you are not certain how to see this info).  Let's ignore for a second that it is possible to have a hybrid token/claims app that uses the IIS web service extension to build a Windows token for you, but also uses the HTTP Module to build a SingleSignOnIdentity so you can read the claims (this is possible!).  You are just really curious about what's in that cookie.  It looks like it contains a big blob of Base64 data.  Can we see the actual SAML token somehow?

Sadly, the answer is "no", not really.

Digging a Little Deeper

At this point, you may choose to stop reading or explore the futility in this a little further.

So, the data is clearly Base64-encoded, but a quick decode of it doesn't give us any angle brackets.  We wonder what gives, so we open up Reflector and start looking at the reverse engineered source code to find out what's up.

As we dig around, we soon discover that the WebSsoAuthenticationModule uses a class called LargeCookieWriter to read the data and decode it.  Interestingly, the Decode method actually calls into an unmanaged code function in another DLL installed by ADFS to do some sort of decompression/unmunging.  Interesting...I wonder what sort of magic goes on in there?  Must be cool if it had to be unmanaged C++!  But anyway, we are not deterred by this as a little reflection code allows us call LargeCookieWriter as well without any details of what's going on down deep.

So, we call the Decode method and we get a string back with readable characters.  Sort of.  Some of the data in there looks like our claims (I see a familar UPN and email address for example), but some of it is Base64 and none of it is XML.  Where's my SAML token???

At this point it gets a little wacky, as apparently there is some sort of XML in this data somehow as ADFS has implemented its own set of XmlReader classes to read this stuff.  This is all handled internally by the constructor on the SamlToken class.  The implementation is a little hard to follow though and we can't easily call the constructor on SamlToken ourselves because it also needs the TrustPolicy and I don't have one in memory right now and I'm getting a little tired of reflection code, so I give up.  Perhaps I'm just too easily deterred, but I really just wanted to see angle brackets and it does not look like there are any to be seen.  Sigh.

The moral of the story is that when the product team says that the _WebSsoAuth cookie is a proprietary format that is not intended to be read without their code, they mean it.  If you want the claims, you are supposed to go through the HTTP Module to get a SingleSignOnIdentity.  This gives them some flexibility to change how things work under the hood in future releases and other products.  Is this a violation of the WS-Fed spec?  Apparently not, as it doesn't specify the token format used in the cookie.

Should the Only Consumers of Claims Data Be .NET 2.0 Apps?

Would it be nice to have a way to get the claims as XML based on the cookie?  I think so.  The real use case in my book is allowing access to the claims for apps that aren't using .NET 2.0.  This could be a .NET 1.x app or could be another app platform altogether running on IIS.  It seems reasonable to me that I should be able to grab the cookie and call a web service method on the federation server to get a nice text-based XML SAML token to examine.  This just opens up the platform to allow a greater ecology of applications to participate.

Another option is to capture the redirected form post from the resource federation server when the user initially returns from logging in.  This does actually contain the XML-based SAML token.  The problem is that this event only happens once and triggers an immediate redirect, so there is an excellent chance you won't be able to grab it.

Still another option (which I think may be the best one) would be for the ADFS IIS web service extension (the part of ADFS that makes the token app work) to populate custom HTTP headers/server variables that contained the claims data.  Then any app platform running on IIS from ASP 3.0 to Rails to consume the claims.  The Windows token would be a secondary consideration, and in this case you could treat that token the same way you treat the anonymous user token in other forms authentication IIS apps (by ignoring it for purposes of authorization).

(ps: if anyone wants to see the reflection code required to actually call the Decode method and get the underlying string data, let me know and I'll post it.)

Authentication Mechanisms in SDS

In ADSI and System.DirectoryServicess (SDS), we are really only given two choices for how authentication will be performed when we do a bind operation.  We can specify the "Secure" flag (AuthenticationTypes.Secure), and ADSI will attempt to the bind to the remote directory using Windows SSPI authentication with the Negotiate protocol.  As you may know, the Negotiate protocol is the primary authentication protocol in Windows since Windows 2000 and selects between using Kerberos (always preferred) and NTLM (there for backwards compatibility).  For the really nerdy of you out there, the Negotiate protocol is implemented by Microsoft's LDAP API using the GSS-SPNEGO SASL mechanism.

The other option in SDS is to NOT specify AuthenticationTypes.Secure and it will attempt to use an LDAP simple bind instead^*.  The only authentication mechanism defined by LDAP specification is the simple bind, so every directory implements it.  As such, it ends up being the cross-platform lowest common denominator approach.  The problem with the simple bind is that it is totally insecure by itself.  Simple bind passes the user's plaintext credentials on the network, so unless some sort of channel encryption is provided (like SSL, the defacto cross-platform standard here as well), anyone who can sniff the wire traffic can recover the user's password.  This sort of thing is generally frowned on my security enthusiasts.  :(

Digest Authentication in LDAP API/SDS.P

There is, however, a richer set of authentication mechanisms supported by Microsoft's LDAP API than what ADSI/SDS get to use, and these are available to System.DirectoryServices.Protocols (SDS.P).  One of the most interesting of these is the Digest authentication protocol.  Digest is a standard, secure authentication protocol that does not involve the exchange of plaintext credentials and is implemented by AD and ADAM, as well as some other LDAP directories.  In SDS.P, we can use it very easily.  Here is a short sample illustrating this (and no, there is no ADAM instance at adam.joekaplan.net; dream on...):

public static bool DoDigestAuth(NetworkCredential cred)
{
    constint LDAP_INVALID_CREDENTIALS = 49;
    LdapConnection conn =
        new LdapConnection("adam.joekaplan.net:389");
    conn.AuthType = AuthType.Digest;
    conn.Credential = cred;

    try
    {
        conn.Bind();
        returntrue;
    }
    catch (LdapException ex)
    {
        if (ex.ErrorCode == LDAP_INVALID_CREDENTIALS)
        {
            returnfalse;
        }
        else
        {
            throw;
        }
    }
}

Here, we have a trivial little sample that shows how this might be done.  Note that we probably wouldn't create a new LdapConnection object each time this is called and we'd also try to find a way to clean that up with a proper Dispose, but the point here is to illustrate that all we really have to do is set the AuthType property to AuthType.Digest and it works.  The credentials are specified with a standard NetworkCredential object.  For Digest, just specify the username and password, not the domain parameter.

Why is This Important?

Digest authentication is really interesting particularly for ADAM, as it is supported by ADAM principals.  Before ADAM supported Digest auth (as of SP1), the only way to authenticate an ADAM principal was with a simple bind.  That wasn't secure (see above), so SSL was required to make it secure.  However, SSL certs are not always easy to procure and take extra effort to install, so many people were missing this and were insecure.  :(

The other cool thing with Digest is that since it is implemented through Microsoft's SSPI model, it can also be used as a mechanism for getting channel encryption and signing, just like negotiate auth supports today^**.  This means that not only can you get secure binding for free, but all of your network traffic after the bind can be encrypted and signed for free (no SSL required).  This is done through setting the Signing and Sealing properties to true on the LdapSessionOptions class.  This, in turn, allows a way to do LDAP password operations on ADAM principals without using SSL and without having to change the setting in ADAM to turn off the requirement for a secure channel on password ops (another big security frown sandwich there...). 

One great scenario for this feature would be to use it with ADFS for their ADAM Account Store support.  This would allow secure authentication without requiring SSL out of the box and would also provide free channel encryption.  Product team please take note!

_{* There is a feature in Windows Server 2003 SP1 ADSI will try Digest auth when the Secure flag is specified and ADSI detects that the server supports Digest but not negotiate auth.  However, this doesn't help us for AD or ADAM, since they both support negotiate auth.  The problem with negotiate auth for ADAM is that it is only used for authenticating Windows users, not ADAM principals.}

_{** There appears to be a bug in the implementation in ADAM SP1 where the channel encryption feature only works if the network traffic is NOT on the loopback (localhost) port.  This is different from the way negotiate channel encryption works.  This appears to be an oversight in the implementation and not an underlying issue in the design, so presumably this will get fixed someday.}

When I was at TechEd this year, I ran into a few people asking how to integrate SecurID authentication into ADFS.  As it currently stands, Microsoft has no direct support for this, or any other authentication mechanism besides AD/ADAM password authentication and client certificate auth.  Hopefully in the future, Microsoft will make the account store and authentication mechanisms a first class extensibility point, but for now, we must hack.  :)

As you can probably tell by the article title, I have gotten this working in my organization's environment, even though it is not supported by MS.  So, how do we make this work?  First, I'll explain the overall principles of how we glue these things together and then I'll share how I actually did it.  In part 2 of the posting, I'll talk about some other ways that we might get this working.

Basic Principles

In Windows Server 2003 and Active Directory 2003, Microsoft implemented an important and useful extension to the Kerberos authentication protocol called "Service for User" (S4U), aka Protocol Transition.  S4U adds the ability for a service to authenticate a client with a non-Kerberos protocol and then transition to a Kerberos-based identity for accessing Windows resources on local or remote services (hence "protocol transition").  Protocol transition is something that can be done automatically by services like IIS when a user authenticates with Basic, Digest or NTLM, but it can also be called programmatically and used by non-Windows authentication systems such as SecurID.

To use S4U, your code must execute on a Windows Server 2003 machine (or higher) AND your Active Directory must be 2003 forest functional level.  If you are doing ADFS, you already have the former.  However, since ADFS supports both Windows 2000 Server and Windows Server 2003 AD, you might not have the latter.  If you don't, you are basically SOL for now.  Upgrade!

When using S4U programmatically, you are basically calling the Windows API LsaLogonUser with the S4U option specified.  When usings the S4U option, LsaLogonUser only requires you to know the user's UPN, not their password, in order to get a Windows logon token for them.  There are some restrictions on how this token can be used, but that really isn't too important for this discussion.  Keith covers this stuff in detail in his book and several articles anyway.  One other nice thing is that .NET (as of 1.1 and higher) has a very easy way to call LsaLogonUser for S4U with the WindowsIdentity constructor that just takes the UPN.  It is that simple.

So, now we know we need to use S4U to get a Windows token for the user and we'll need their UPN to call this.  How does the Windows token then help us with ADFS?  Well, the ADFS logon service (LS) has a method that allows a user to authenticate given their Windows token.  In fact, the /ls/auth/integrated directory on the federation server does just this.  It is set to use Windows Integrated authentication (IWA) in IIS.  IIS logs the user on with IWA, passes the corresponding Windows token to ASP.NET.  Then, the ADFS HTTP Module grabs that and calls the correct method to log you into ADFS and now you have a federation token. 

An interesting side note is that ADFS itself uses protocol transition in the web agent for token-based applications to achieve similar things (assuming again that you have 2003 AD; otherwise it uses its custom authentication package). 

So, our basic approach with SecurID authentication is to:

1. Authenticate the user with SecurID using one of RSA's supported methods for web authentication
2.  Get the user's UPN somehow (perhaps an LDAP query?)
3. Use S4U to get a Windows token for the user
4. Use the appropriate method on the LS to authenticate with ADFS using the token

This is cake!  :)

How I Did This

Ok, I cheated a little bit.  In addition to using RSA's SecurID/ACE Server product in our company, we also use their ClearTrust web SSO product (now apparently called Access Manager, but I didn't get that memo).  ClearTrust already basically let's me do steps 1-3 above using their standard product when the protocol transition/S4U feature is enabled.  Since it runs as an ISAPI Filter/ISAPI Extension (via a wildcard map in IIS), it runs before any .NET code executes.  All I had to do was configure ClearTrust on my /ls/auth/integrated directory, and ADFS doesn't know the difference between IIS having used IWA or some other thing.  The LS just grabs the Windows token from where it expects to find it in ASP.NET and proceeds with logon.  It really is cake and actually worked the first time I tried it.

The hardest part was instructing ClearTrust to NOT execute on any resources other than the /ls/auth/integrated directory, which essentially came down to a lengthy URL exclusion list in the ClearTrust configuration.  Anyone who has used ClearTrust probably already knows how to do that.  I will probably experiment with alternate directory structures to simplify this a bit more in the future.

Since ClearTrust is a web SSO product, it also contains a signout function that is implemented by navigating to a page.  We integrated this into the ADFS signout by using the same technique it does: we referred to the ClearTrust signout page in a hidden image tag on the normal signout page.  That results in our ClearTrust cookie being cleared.

One thing I should make clear is that we did this customization on the FS, not the FS-P (Federation Service Proxy).  I don't see a reason why you couldn't do this there too, but you'd need to mess with it a bit more as the FS-P is configured out of the box to do its own forms auth instead of redirecting to the /ls/auth/integrated directory.

One important caveat with SecurID auth is that it basically requires forms authentication, especially to support PIN operations.  This basically implies that you can't use the "basic" authentication support built in to ADFS here.  As such, you should probably turn off that option in your ADFS configuration.  That may also mean that you some SharePoint/Office integration stuff breaks though.  I'm not sure how to reconcile that problem.  Forms auth and Office don't like each other.

Summary

So anyway, that's the basic concept.  If anyone needs additional details, let me know and I'll try to expand on this.

In Part 2 of this post, I'll try to expand on how one might accomplish if one were not so blessed with a working ClearTrust infrastructure in place to do the hard part.  Note that I've never actually done this part, so some of it might just be wild supposition and it might take some help from someone else out there who really wants this to make it happen.  We'll see.  In the meantime, I hope the blueprint was helpful.

One of the things you run into with ASP.NET apps doing authorization using Windows security is that you often need a way to find out the authenticated user's identity and security group memberships for troubleshooting.  This is especially useful under ADFS, where your Windows token can go through a mapping process based on claims received from an external organization and bear no resemblance to an actual user in your AD forest.

This is just an ugly sample page in ASP.NET (using VB.NET, but I'll do C# on request if that's really important; we're talking about 10 lines of code here guys...) that dumps out the authenticated user's groups and name.

The core function looks like this:

Private Sub Page_Load( _
    ByVal sender As System.Object, _
    ByVal e As System.EventArgs _
    ) HandlesMyBase.Load

_nameLabel.Text = User.Identity.Name
Dim groupSidHtml AsNew System.Text.Stringbuilder
Dim sids As System.Security.Principal.IdentityReferenceCollection = _
    DirectCast(User.Identity, System.Security.Principal.WindowsIdentity).Groups
Dim names As System.Security.Principal.IdentityReferenceCollection = _
    sids.Translate(GetType(System.Security.Principal.NTAccount))
ForEach name as System.Security.Principal.NTAccount In names
    groupSidHTML.AppendFormat("<p>{0}</p>", name.ToString())
Next

_groupLabel.Text = groupSidHTML.ToString()
EndSub

If I were a little less lazy, I probably would have added the imports declarations on the page instead of using the full type names and would have used a repeater and some formatting, but this was quick and dirty.  Feel free to improve it.  The working page can be downloaded at the link at the bottom of the page.

Caveat

Even though token apps allow you to run on prior versions of the .NET framework, this page uses a bunch of .NET 2.0-specific code in it (IdentityReferenceCollection and such), so you must configure the app for .NET 2.0 to use this.  I'm simply not at all interested in writing all the p/invoke stuff to crack the user's token and translate their SIDs into names simply to create a .NET 1.1 solution.  Sorry.  There is lazy and then there is just wasting time...

Also, you must be configured for Windows authentication in ASP.NET, but that should be obvious I hope.

I hope this helps someone figure out what ADFS is actually putting into their Windows token!

(Update, changed the file to a .zip to avoid error mentioned in the comment)

One of the tricks you can do with the Active Directory Federation Services (ADFS) home realm discovery process is get a user to skip the home realm discovery page completely if you embed a query string in the application URL that tells ADFS what realm to use.  The query string is:

whr=xxxxxx

where xxxxxx is the federation URI of the partner (which they tell you when you set up your federation, or you create when you are setting up your test lab).  That typically looks like:

urn:federation:myorganization

Thus, the whole url might look like:

http://www.joekaplan.net/?whr=urn:federation.myorganization

(no, this site is not federation-enabled and won't be any time soon...)

Using these home realm query strings is very handy, not only for getting your own organization's users to the target app quicker by allowing them to skip a page that may potentially have many choices, but also just for testing.  The query string overrides the persistent cookie you may have that identifies your home realm, so you can use this to avoid having to delete your cookies all the time.

However, if you want to refer to the resource partner's account store with this trick, you don't use the resource partner's federation URI.  Instead, you use the "built-in" URI:

urn:federation:self

I'm sure this is probably documented somewhere (or maybe not; the ADFS docs have a ways to go...), but I had to figure it out the hard way and I thought I'd share.

The Problem

So, let's say you are building a claims-based application for Active Directory Federation Services (ADFS) and you want to use VS.NET 2005 to do this.  Alternately, let's say you are customizing some of the built in pages that come with the federation server or federation server proxy and want to use VS.NET for that.

As things stand today, there is a minor annoying friction point here as the ADFS installer doesn't provide a nice clean way to set a reference to the code you need in System.Web.Security.SingleSignOn and such.  Even if you are developing on a 2003 box (which a lot of serious web devs do, although I still stumble along with XP as its the company standard) and actually install ADFS, you don't see these assemblies in the .NET tab on the add reference UI.  The problem is that the ADFS installer puts the assemblies directly in the GAC and doesn't leave a copy on the "normal" file system (which the compilers actually need), nor does it bother to create a registry key under:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v2.0.50727\AssemblyFoldersEx

(which is the little bit of magic that gets your assembly in the .NET tab).  Multiply this by a large team working on a claims app, and this can be a drag.

The Solution

So, what's a dev to do?  Well, you can definitely copy the assemblies yourself, GAC them and create a registry key and probably do all that with a little batch file if you are so inclined.  Or, you can go over the top and create your own MSI installer package that does this for you.

As a little experiment to see if I could do this quickly and learn how to do some new stuff in MSI that  thought I'd need later on, I did the latter.  It took about an hour over lunch one day.

I'd love to just distribute the msi package directly, but I won't consider doing that unless MS gives me permission to do so (which I don't expect to get).  So, that won't happen.  However, I will tell you how to build your own (fairly) painlessly.  For this, you will need:

Ingredients

• One copy of Windows Installer XML (WiX) 2.0 (3.0 might work as well, but I didn't bother trying)
• One copy of the ADFS.msi package that comes with R2 (dig around on the install media for it), or alternately, the 3 assemblies that come with ADFS that are installed the GAC that you hoisted from a normal install
• One copy of my sample WiX file that will create the MSI (see link at bottom)

Recipe

1. Place the vsadfs.wxs file in a new clean directory on your file system
2. Use dark.exe from the WiX distro to reverse engineer ADFS.msi in order to get its files into a handy .cab format, ignoring the error from Dark (<wixpath>\dark -x bin -out adfs.wxs adfs.msi)
3. Grab the three assembly files (S.Web.Security.SSO *) from the cab file and stick them in a directory called "files" underneath the directory where you put my vsadfs.wxs file
4. Run these two commands from the command prompt, filling in the path to your WiX binary install where appropriate (unless you have that on your path, obviously):
   1. <wixpath>\candle vsadfs.wxs
   2. <wixpath>\light -out vsadfs.msi vsadfs.wixobj <wixpath>\wixui.wixlib -loc <wixpath>\WixUI_en-us.wxl

You now have a working msi installer that will install (and uninstall and repair!) these things for you.  You may notice that the UI thing I used, WiXUI, shows a license screen and shows the CPL license there.  That's because we didn't supply a license.rtf file in our directory, so WiXUI picked up the one that comes with WiX in its directory.  Feel free to change it.  If you want to eliminate that dialog completely, you are on your own.

Caveats

• This will NOT give you a working version of an ADFS-enabled web server!  You cannot use this to actually generate a SingleSignOnIdentity object to be used for testing.  You would need some sort of a mock object thingy to do that for you (may that's next?).  All it really does is let set a reference in VS.NET and compile the code, but that's better than what you had before.
• Additionally, my current installer uses different component GUIDs for the version of the files that go in the GAC.  I'm not sure if it would be the right thing to reuse the component GUIDs from the real ADFS install which puts these components in the GAC or not.  The only time this might be an issue was if you ran this installer on a box that already has ADFS installed.  I'm not sure what would happen, but right now it seems better to keep them separate. 
• The installer allows specify the install location if you want to change it and breaks the VS.NET and GAC install stuff into separate features so that you don't have to install both if you don't want.  This can be customized via the feature tree dialog.
• The installer does not bother to check if you have VS.NET 2005 or even .NET 2.0 installed.  This would be a good sanity check for mass market usage, but I'm hoping the users of this are not too insane. 
• The installer defaults to a "per-machine" install instead of per-user.
• Repackaging Microsoft's installer is probably a violation of something and may void your warranty or get you in some sort of trouble that I had not fully considered when offering this suggestion.

The Future

I'm hoping that future versions of ADFS will have this scenario already considered so that things like this won't be necessary.  Perhaps we can convince the team to ship this as part of the product or as a free download/sample solution?  I wouldn't be surprised to see these assemblies in a future version of the .NET Framework as well, which would mean that we'd get this for free another way.  Until then...

(Updated 2 Aug 2006 changed to zip file)

I've been telling myself I was going to finally get into the blogging racket for almost 2 years now, but even though I've had the hosting all put together for that entire time, it took me forever to actually get it together.  I just had to do it myself instead of using another site...

Anyway, this blog will probably resemble the blogs I already read, in that it will be mostly technical with a focus on building software using Microsoft's .NET platform.  It will probably lean heavily on my specialties, .NET LDAP programming and application security, but will likely also feature other stuff I'm into like application architecture, agile development, identity federation, cryptography, and setup development in MSI using WiX. 

Speaking of .NET and LDAP, if you've ever heard of me before, it is most likely because you might have stumbled across one of the myriad usenet posts I've made over the last 4-5 years on the Microsoft newsgroups, or perhaps I answered your question directly.  Micrsosoft has actually designated me an MVP in this area, and I've even written a book about this with my intrepid co-author, Ryan Dunn, to further our aim of providing resources for the .NET community in this obscure, but stranglely difficult and increasingly important aspect of software development.

That's all for now.  Maybe some real content next time, eh?