Ubuntu 12.04 deployment with Active Directory

** Update 4/13 – Added winbind. Winbind is now my preferred solution.

This is a follow up to my past post. I want to deploy 12.04 for what I consider a typical enterprise environment. That means centralized authentication, file shares, roaming profiles, etc. Your first step should be to acknowledge this is a very hard project to do. If you need to deploy some computers this week, go get your Windows install CD right now. Linux takes about 10 100 times as long to set up.

Authentication to Active Directory

Options include Centrify, likewise open, winbind, and pam_ldap. They are suck terribly. My opinion on which is the the best differs often. Right now I’d say winbind. If you just need a quick solution just for your own use, likewise works and is much easier. Here’s a quick comparison:

Winbind Lightweight and fast. Terrible documentation. Hard to set up.
Likewise open Easy and includes a GUI. Causes slow log in times.
Centrify Very buggy, would not recommend.
pam_ldap Both buggy and hard and lacks features.

There is also sssd which I have never been able to get working. You can read this post in the Ubuntu-Enterprise list server.

Either way they both suffer from a fatal flaw that would probably keep all but the most dedicated linux sysadmins far far away from Linux. They don’t work in a real networked environment! I’ve talked about this issue before, basically lightdm starts up before networking is up in a race condition. The user types in a name, password, gets an error, calls tech support and has the impression that this “Ubuntu” is quite the useless operating system.

First with either option you need to allow users to type in a login in lightdm. Edit /etc/lightdm/lightdm.conf and make it so users can type in a username by adding

greeter-hide-users=true

Also make sure your wireless network is available to all users. (Click the network icon, Edit Connections…). Next follow instructions for either Likewise or Centrify.

Likewise Open

It’s terribly easy to install it. It even has a GUI. So I won’t include instructions for this. However if you need it working on boot as I described above do this. Place this script somewhere like /opt and make it executable. It just checks to see if any networking is up or time out at 20 seconds. You may need to adjust it or even place a sleep statement after it. Just experiment on your hardware/network.

#!/bin/bash
i=0
while [ $i -lt 20 ]; do
 sleep 1
 is_up=$(ping -q -w 1 -c 1 `ip r | grep default | cut -d ' ' -f 3` > /dev/null && echo 1 || echo 0)
 let i=$i+1
 if [ $is_up -eq 1 ]; then
 let i=999
 fi
done

Now edit /etc/init/lightdm.conf and add under the emits section. This script just runs ping test before starting lightdm then waits 5 extra seconds to give things time to settle. You should test this our throughly and make it longer if you have authentication issues.

pre-start script
 /opt/pingtest.sh
 sleep 5
end script

winbind

Ubuntu community wiki will give you information that may or may not be vaguely related to a bunch of terms you may or may not understand. Let’s start with some definitions to be clear what we are doing.

Kerberos – A system that let’s users log into services like share drives without retyping the password. Basically it’s some crap you need to set up.

win_bind – This does the actual authentication. It’s part of Samba.

Install these packages

sudo apt-get install winbind samba smbfs smbclient krb5-user

The krb5 installation will ask you for a realm. If your domain is ADMIN and your network is something.com then your realm is ADMIN.SOMETHING.COM. I didn’t have to tweak the krb5 conf file really. Test it with knit some_username. If it doesn’t work experiment with re-configuring it using  dpkg-reconfigure krb5-user. I didn’t do this stuff at all and don’t understand what it is.

Next set up samba by editing /etc/samba/smb.conf

Here is my smb.conf just replace the domain with your own. Who knows what all that does. You can read about it more here. Test your smb.conf by running sudo testparm

To “Join the domain” run

sudo net ads join -U Administrator

You will get a DNS update failed but that’s just a trick to make you think you screwed up the configuration.

If you want users to be able to change their passwords, check out this bug.

Do you like having home folders? Well the default now is not to create them. That sure makes sense. Add them back.

Now run sudo pam-auth-update to update pam configuration. I strongly suggest ensuring pam-auth-update works and that you don’t manually hack the pam.d files. On an update your users might unknowing blow away your hacks.

Test the set up but be aware it probably won’t work on boot until you log in as someone else. This is because of the network race condition issue. See the part about pingtest in the likewise open section. Do the same thing. I also discovered that when winbind is in the unworkable state where you can’t log in, running wbinfo -u fixes it. So I made my lightdm init script like this instead. Yay hacks.

pre-start script
 /opt/pingtest.sh
 /usr/bin/wbinfo -u&
end script

If you have been toying with other authentication mechanisms your /etc/nsswitch.conf might be off. See here. nsswitch stands for Name Service Satanic Witch – who wants to curse you. You must satisfy the witch in order to proceed.

Centrify

In my experience in July 1012 Centrify will cause odd networking issues. So I don’t suggest it. If you want to try anyway:
You can get it from the Ubuntu partner repo’s. If you don’t have this, you need to check this off in Ubuntu Software Center, Edit, Software Sources. Then update then install centrifydc. Join your domain like

adjoin -w mydomain.org

Centrify doesn’t play nice with pam-auth-update so create a conf file for it called /usr/share/pam-configs/centrifydc

Name: Centrify DC
Default: yes
Priority: 257
Auth-Type: Primary
Auth:
        [success=end default=ignore]                            pam_centrifydc.so try_first_pass
Account-Type: Primary
Account:
        [success=end new_authtok_reqd=done default=ignore]      pam_centrifydc.so
Session-Type: Additional
Session:
        required                                                pam_centrifydc.so homedir
Password-Type: Primary
Password:
        [success=end new_authtok_reqd=done ignore=ignore default=die]   pam_centrifydc.so try_first_pass
Password-Initial:
        [success=end new_authtok_reqd=done ignore=ignore default=die]   pam_centrifydc.so

Now run pam-update-auth and centrify will play nice with other modules such as pam-mount

Next we need to resolve the can’t log in on first boot problem. In likewise we just made a simple is networking up yet test. This doesn’t work with Centrify. From what I can tell this is what happens.

  1. Networking starts
  2. pingtest script launches lightdm only after networking is up. Right now we could ping the domain controller
  3. Centrify attempts to contact the domain controller but fails for some unknown reason. It then sites and does nothing for a bit. Then tries again and succeeds. This takes about 30 seconds.
Instead we need to modify /etc/init.d/centrify and add some upstart emits. These will tell us when centrify is ready to use. Look in the start section and add this emit as shown in bold
start)
 adclient_check
 echo -n "Starting $NAME: "
 start-stop-daemon --start --quiet --exec $DAEMON --pidfile $PIDFILE \
 -- $OPTIONS
 RETVAL=$?
 if [ $RETVAL -eq 0 ]; then
 echo "OK"
 wait_adclient
 # upstart won't start gdm until we say we're connected
 initctl emit centrify-connected # added
 else
 echo "FAIL
This emit signal doesn’t do anything in itself. It’s just an upstart signal we can look for else where, such as /etc/init/lightdm.conf where we will add it at a dependency to start.
start on (filesystem
 and started dbus
 and (graphics-device-added fb0 PRIMARY_DEVICE_FOR_DISPLAY=1
 or drm-device-added card0 PRIMARY_DEVICE_FOR_DISPLAY=1
 or stopped udevtrigger)
 and centrify-connected)
stop on runlevel [016]
But what if the user is at home and will never connect to the domain controller? Well it will take a long time to boot. I’m sorry. If you have a better solution please comment. Let’s compare are boot time to see if it’s reasonable. All tests are on a Lenovo x120e. They are from power button to login prompt
Ubuntu 12.04 with my modified init scripts.
53 seconds (28 of those seconds are just waiting idle for Centrify)
Windows 7
47 seconds
Windows XP
A blazing 26 seconds! It would be pretty hard to recommend Linux if MS didn’t screw up NT with Windows Vista/7.
Anyway I’ve determined we are almost as fast as Win 7 which is what a x120e comes with. While it makes me cringe that I’m making the computer even slower, it’s on par and that makes it a good enough solution. Factor in time waiting for Win 7 to be generally slow after login and one could argue Linux is just as….slow. As much as I hate Linux, I hate Windows more.

Wifi for non admin users

The default in Ubuntu is that this can only be done by admin users. Linus Travolds himself actually ranted about this policy in opensuse. Here is the fix.

Network users can’t add printers

By default a network user won’t even be able to add a printer. Add all users to the lpadmin group using this.

Make updates not replace conf files

By default Ubuntu asks users if they want to replace configuration files on updates. A more honest dialog would say please click a button, one will break your system while the other lets the update continue.

To disable this – create the file /etc/apt/apt.conf.d/local

Dpkg::Options {
 "--force-confdef";
 "--force-confold";
}

An addition step would be to enable unattended upgrades. This will force most updates without user intervention. That scares me a little. Here are instructions on enabling it. Notice the option to break down upgrades into small chunks so that shutdown is still possible without too much delay. No Windows style 30 minute shutdowns hurray!
https://help.ubuntu.com/community/AutomaticSecurityUpdates

Desktop Environment

People seem to like Unity these days, but I still don’t. It’s also unfamiliar with new users. I’m more concerned about reducing tech support calls than wowing users. It’s fairly easy to remove it (but why no gui way?). Install gnome-fallback-session. Now edit /etc/lightdm/lightdm.conf and set

user-session=gnome-classic

Log in as a new user to test, to ensure it’s not just using the previous session. I like to just have one bottom gnome-panel sort of like classic windows.

Also install compiz config manager and enable alt tab which is notably missing! And once more I suggest playing around with compiz on your computer model. It’s often can be very buggy and you may want to just fully disable it.

Copy your ~/.config file into /etc/skel but then delete anything you don’t want (firefox, chrome, etc) so that new users get the same configuration you have now. There used to be a program called sabayon that did this in a more user friendly way but it’s too buggy and it’s faster to just copy things to /etc/skel. All in all the Windows way of having All Users is much easier IMO.

Also for some reason I don’t quite understand you need to follow this to make /etc/skel copy files to the new users home. What I don’t get is that they seem to copy on a vanilla ubuntu client but as soon as you set up pam_mount they just stop working without this. Anyway copy the file and run pam-auth-update and it works.

Centralized Printing

You can control printing by having a cups server.  Run system-config-printer and click Server Settings. Check Show printer shared by other systems. Hit Ok. Now Click Server settings again (this is a bug). Now advanced. Now click add and type in the address of your cups server. The printers will just show up like magic! If users aren’t allowed it to print it just won’t print. Which is really annoying actually. Why can’t it prompt for a password? It used to try this but the prompt was broken and never actually worked :(

I submitted a bug report about it not asking for credentials.

I’m tempted to try out samba4 printing for Windows clients. In my experience it does a poor job except in trivial configurations. I wasn’t able to list printers in the directory for example.

Windows Applications

There’s probably a lot of Windows only applications you need to run. Crossover/Wine work for some like Office 2010. It’s really really buggy though. See my experience here. For programs that won’t run in wine you can use rdp. Ulteo makes a cool platform that lets users launch apps from a website. Great for Internet Explorer and centralized systems like proprietary databases. However it’s a pain to set up, so be prepared!

Java

Update – I find this the easiest method.

In short – It’s just about impossible to install Java now. Really I have no perfect work around. Maybe you can get away with OpenJDK which runs about 0 out of 10 applications I use. Canonical might just take away your java without warning. You could install it from Oracle…but your users won’t get updates this way. You could try various PPA scripts but it seems Oracle is actively trying to stop this. With the Oracle installation you are left wide open to exploits without updates. Lovely. You’re an IT person if you’re reading this. Do me a favor and DON’T EVER BUY ANYTHING FROM ORACLE.

File Shares – Samba

Samba works with the gui but you still might prefer to use pam_mount. I already talked about it in my previous post so I won’t again. It’s better if you want shares to come up automatically for users. If you just have a few users they can use nautilus to find the shares. Just type in smb://yourserver/share and save it as a bookmark. There is a browse share feature, but I’ve never seen this work in my life. But beware the bookmark thing won’t appear in wine! So if you need MS Office you users won’t be able to save as into the share!

If you do use pam_mount make sure to work around this bug that prevents likewise-open from working with it.

Update – pam-mount with a fast computer bug – On a modern computer a user might be able to log in before networking is up. If you use the pingtest.sh I described above you won’t be affected. However if the user logs in before networking is up, shares won’t mount. Obviously pam-mount will give up instead of retrying because it’s utter garbage and a better name might be pam-do-nothing.

Rant – Mounting a file share sounds like the most basic enterprise function, but Linux has no good solution. The GUI way doesn’t work in all applications. Pam mount is not ideal because if you turn on the computer without networking (at home?) it requires a log out (or terminal su) for it to work. Also pam_mount rarely umounts correctly which can cause odd delays sometimes. It just sucks.

Imaging

I like to use clonezilla for imaging. Here is a script you can use to automate AD joining. Take a minute to look this over and put in your info. Then run it on cron @reboot. It runs only if it sees the hostname as image. Note this script is for likewise open. For centrify just replace the join command.

#!/bin/bash
hostCurrent=$(hostname)
hostOld='image'
if [ "$hostCurrent" == "$hostOld" ]
then
 date > /opt/ad.log # overwrite log the first time around
 # DOE mangled machines to the point that the first serial number is blank!
 host1=$(/usr/sbin/dmidecode | /bin/grep -E 'Serial Number:[[:space:]]*[^[:space:]]+' | /bin/sed 's/.*: \(.*\)/\1/;q')
 host=$(echo $host | /bin/sed 's/[ ]*//g')
 hostname $host
 echo $host > /etc/hostname
 /opt/pingtest.sh
 sleep 10
 (
 /usr/bin/domainjoin-cli join your.domain.org 'user' 'password'
 adreturn=$?
 ) 2>&1 >> /opt/ad.log
 if [ $adreturn -ne 0 ]
 then
 echo "adjoin failed." >> /opt/ad.log
 exit 1
 fi
 echo "I'm itching to reboot." >> /opt/ad.log
 /sbin/reboot
fi

I made this to take inventory. Sorry it sort of sucks and has no installer. Basically the idea to that a script runs that feeds some stats like hostname and hd freespace to some database.

Admin access

If you want to give some users sudo edit /etc/sudoers and add something like.

%ADMIN\\UnixAdmins ALL = (ALL) ALL

But note that there is a bug that prevents it from really being useful. This will let users do sudo on the command line but not in the gui. The work around is to just manually add each user to the local sudo group. Have fun with that. Comment if you have a better way.

File Sync (roaming profiles)

You can try using unison. Owncloud has a desktop sync tool but it’s still in beta sucks. I’m using that unison script for now. For desktops just use samba instead since you don’t need the files to copy to the local hard drive.

Web Browser

I use Chrome as the default web browser. There are some not so well documented ways you can make it a nicer experience with default preferences. I really hate the many pop ups it has on first launch for instance. First read this
http://www.chromium.org/administrators/configuring-other-preferences
I don’t think it does a good job explaining that master_preferences goes next to the google-chrome executable (i.e. /opt/google/chrome/master_preferences)

The real exhaustive list of preferences is here right in the source:

http://src.chromium.org/viewvc/chrome/trunk/src/chrome/common/pref_names.cc?view=markup

‘user_skipped’ disables the sync promo for instance. Here is my master preferences file. Over 9000 promo views is just for fun.
{
 "homepage" : "companyportal.page.com",
 "homepage_is_newtabpage" : false,
 "browser" : {
 "show_home_button" : true,
 "check_default_browser" : false
 },
 "distribution" : {
 "skip_first_run_ui" : true,
 "show_welcome_page" : false,
 "make_chrome_default" : false
 },
 "first_run_tabs" : [
 "http://www.cristoreybrooklyn.org/portal"
 ],
 "sync_promo": {
 "startup_count": 1,
 "user_skipped": true,
 "view_count": 9001
 }
}

After install configuration changes

I use puppet, see this post. If you have time set this up before deploying you will save yourself a lot of time later. You might even consider placing a lot of the hacks I mention in puppet to ease future deployments.

Overall thoughts

12.04 IMO is as many steps forward as it is back. Some bugs are fixed and some are introduced. It’s a perfect example of why corporations don’t use linux outside IT. You can spend weeks setting up the perfect image in 11.10 but then 12.04 comes out and nothing you did before works. Here are my thoughts overall

+ Much better battery life
+ Gnome fall back session is a little less buggy.
- Gnome Classic is not as good as 10.04. For example the missing alt tab and in my experience compiz is less stable in Gnome Classic. Also sometimes wifi manager doesn’t display right for me. Again this is a trivial issue but for an end user it means calling tech support and for a company it means Linux is wasting money.
- Likewise open is more buggy (doesn’t work with /etc/skel)
- CUPS is more buggy (no warning about authorization failure)
- Less configurable overall compared to gnome 2.
- Not Ubuntu’s fault, but java is now a nightmare

My advice to Ubuntu is to focus on quality control instead of UI. But it looks like they are going after consumers and not businesses. If they just fixed the bugs I mentioned I would recommend Linux to any small business. As is, I would only recommend it to companies that have a deep understanding of Linux already. Windows Server will continue to dominate until Ubuntu is Linux for human beings who have jobs.

As a LTS release 12.04 disappoints me. After all the configuration work – the end user gets a worse experience in my case. A longer boot time and cups is less clear about authentication. But 11.10 is not LTS and not viable for long term deployment. My best hope is that some bugs will be fixed after release but based on past experience this is highly unlikely.

25 thoughts on “Ubuntu 12.04 deployment with Active Directory”

  1. > They don’t work on wifi!

    If this were true I would be unable to log onto my wifi-only networked laptop using CentrifyDC. Which I do.

    1) Centrify & Likewise both cache credentials.

    2) You can configure the Network Manager to connect to a network *before* logon occurs. As long as, when the user attempts their connection, they have a viable network connection and are ad-joined, the logon event will work. (And once they’ve logged on once, the local caching will work thereafter normally.)

  2. On some networks, mDNS is messed up. Edit /etc/nsswitch.conf and remove the mdns service specifications and result reactions like so:

    hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4

    becomes

    hosts: files dns

    This’ll help if stopping avahi aleviates name resolving issues.

  3. They aren’t tested with Samba and you would probably need to run Samba 4 which isn’t stable. It would be nice to have an easy way to join a Samba PDC. My experience with pam_ldap, winbind is that it’s highly complicated and frustrating. I ended up using just pam_ldap but getting every out of the box feature in likewise/centrify like cached credentials and groups is a pain. It’s funny how much easier it is to get Linux to join a Windows domain than any Linux equivalent.

    I want to play around more with Zentyal which seems to have a desktop client that can join a Zentyal Samba/OpenLDAP server.

  4. can anyone help me figure out how to connect my ubuntu machine to the domain of my company?
    tried centrify and likewise and no success.

    thanks

  5. Centrify does not copy the full /etc/skel

    I’m having the problem in copying files from directory “/etc/skel” for users in AD. For local users the copy its done normaly. You can help me?

    Tanks

  6. I currently use Zentyal at our workplace and we have all the desktops & laptops running 12.04.

    I am eagerly awaiting Zentyal 3.0 (although I dread to accept the sluggish web interface).

    Perhaps I may just create my own Django interface to Samba4 when it comes.

    At the moment I am using Cobbler and Puppet for automated deployment and post configuration and maintenance. This part is actually very nice.

  7. one way that i found to run windows apps in linux without having to deal with wine is to install a vmserver on a network location and to push a unity mode to the linux desktop

  8. Dont compare Ubuntu with MS-server, instead compare commercial products of linux, such as SUSE server and redhat server, with the MS server.

  9. ” Your first step should be to acknowledge this is a very hard project to do. If you need to deploy some computers this week, go get your Windows install CD right now. Linux takes about 10 100 times as long to set up (though easy to clone).”

    This is so true… I really like linux and the whole open source concept but marrying this to Active Directory is just not fun. In terms of analysing things from a business perspective evaluate your time to deploy these devices and your confidence that they will continue to work. Despite it’s many flaws, expense, and vulnerability to viruses etc; Windows just works. Very few worries about drivers and software and compatibility and consistency.

    If you want to build a linux only server environment for file and print sharing to a Windows client environment then Samba is a good route. No Windows OS expense and no CALs. Very low cost.

    btw.. Ubuntu is a commercial version of linux. Unlike other distros that have a commercial version and a development version Ubuntu does not have two distribution streams.

  10. hi,

    I have ubuntu 12.04 with likewise open 6 on windows 2003 server. I can login users in domain, but i cannot auto mount their shares ao login.

    I have pam_mount.conf.xml configured:

    But pam_mount is trying to mount everytime with lighdm username.

    Aug 22 10:03:59 ubuntu- lightdm: command: ‘pmvarrun’ ‘-u’ ‘lightdm’ ‘-o’ ‘-1′
    Aug 22 10:03:59 ubuntu- lightdm: pam_mount(misc.c:38): set_myuid: (ruid/rgid=0/0, e=0/0)
    Aug 22 10:03:59 ubuntu- lightdm: pam_mount(misc.c:38): set_myuid: (ruid/rgid=0/0, e=0/0)
    Aug 22 10:03:59 ubuntu- lightdm: pam_mount(pam_mount.c:440): pmvarrun says login count is 0
    Aug 22 10:03:59 ubuntu- lightdm: pam_mount(mount.c:749): going to unmount
    Aug 22 10:03:59 ubuntu- lightdm: pam_mount(mount.c:647): Could not get realpath of /home/likewise-open/TIC/lightdm/Documentos: No such file or directory
    Aug 22 10:03:59 ubuntu- lightdm: pam_mount(mount.c:218): Mount info: globalconf, user=lightdm fstab=0 ssh=0

  11. Hi David:
    Following your instructions I managed to integrate Ubuntu 12.04 in Windows Server Active Directory. I have used Likewise-Open. Shared drives in AD are mounted perfectly correcting the bug you mention in the previous post. Unfortunately /etc/skel not work.
    Using Remastersys I have created an own distribution. This distribution is integrated when installed. I am integrating 500 computers of Antigua-Luberri secondary school in Basque Country. Thank you very much.

  12. That’s odd about /etc/skel. I saw it happen to me and someone else reported an issue. You should comment on this bug I submitted
    https://bugs.launchpad.net/ubuntu/+source/libpam-mount/+bug/992101
    I don’t know what is making it not work. It does work for me now. I doubt that bug will get any attention so you are likely on your own. Maybe run each step and try to isolate the step that makes it stop working. Then play around with settings. Try a different pam_mount config for instance or something in /etc/pam.d/

  13. We are using sssd for AD joining, perhaps I’ll add more details on the Ubuntu Enterprise wiki soon.

    Anyway, likewise open does not let you share UIDs between machines. Never actually tested Centrify, but would like to see an issue list for people to be able to compare. Perhaps you want to join us on the Ubuntu Enterprise mailing list?

  14. Hello there! This is my first visit to your blog! We are a
    group of volunteers and starting a new project in a
    community in the same niche. Your blog provided us useful information to
    work on. You have done a wonderful job!

  15. I’m loving the pam config you wrote for Centrify – now pam-auth-update works AND pam mount suddenly does what it should :)
    Thanks very much

  16. Hi,

    You make a number of valid points. It is a headache and I am betting everything I’ve done on 12.10 will break with the next release. Oh well.

    I am wondering if you have any thoughts on my last hurdle. I am attempting to create a multi-user disk-less workstation environment where all workstations will use the same image from the same tftp boot server and use Windows active directory log-in credentials. It appears that multiple local users can log in from 2 or more stations without any issues. However, when using active directory credentials, logging in to the second workstation breaks the connection to the domain controller. Apparently there are some files being shared and clobbered. There are some sockets in /var/lib/pbis that I imagine are critical. I assume I need to isolate shared files in a tmpfs.

    Is a multi-user disk-less active directory workstation environment achievable? If so do you have any recommendations on how to prevent the active directory connection from breaking?

    I am using:

    Ubuntu 12.10
    Lubuntu desktop
    PBIS Open 7.0.6.935 – Stable Release

    The above was my question posted over on AskUbuntu.

    Thanks, PLA

  17. In my experience, I’d really advise against a non LTS release. There may be some specific hardware or some reason you want 12.10 I’m sure.

    I’ve done LTSP with Samba (no active directory) before. I had a LOT of issues with the same user logged in multiple times from different thin clients, I would really say just don’t do it if possible. A lot of applications just can’t handle it. Even if you get authentication worked out you will have issues with a number of applications. At least I did.

    Sorry I’m not more helpful in actually answering any of your questions.

  18. David,

    I wanted to stick with 12.04 LTS but support for the Intel HD 4000 GPU seems better on 12.10. The disk-less workstation will be used for web browsing with Chrome and RDP to Windows terminal servers. Neither of those have exhibited any multi-user clashes. I’m going to continue hacking on the active directory multi-user clash for a bit more. Thanks for the great blog post and insight.

    Thanks, PLA

Leave a Reply

Your email address will not be published. Required fields are marked *

CAPTCHA Image

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>