We are experiencing technical difficulties

By Krista Olmstead - February 18, 2009

Dalhousie email has been having difficulties. We understand the current level is insufficient to conduct business. We regret the problems and would like to shed some light on what is happening.

Email traffic has increased significantly since Christmas. Although some was anticipated, traffic exceeded expectations. To deal with this, a previously planned major upgrade has been moved ahead of schedule. The problems may continue for this week, but additional server capacity will be added this weekend to remedy the situation. We have postponed some of the university’s regular mass emails to be delivered at night instead of during the busiest times of the day.

In addition to this, over the next two months, we will continue to add capacity to the service to meet current and future needs of the university. Additionally, we’ll be engaging the campus community to explore alternative email and communication tools.

In the mean time, there are a few things everyone can do to help all of us get through the next week. The greatest load on the system appears to be between 10 a.m. and 2 p.m., at the highest email peaks. For this week only, if you can avoid email during those times, this will help. Also for this week, we will be turning off service to Blackberrys during the 10 a.m. to 2 p.m. period.

We are aware that email is crucial to your work. It is our highest priority to restore email to optimal conditions.

Krista Olmstead handles communications and marketing for Information Technology Services (ITS).

Readers Say

In addition to the details above, I will provide some specific technical details of what is being done for those who want specifics:

1. Our current email server is a 4 dual threaded cores running at 2.7 Ghz with 16 MB RAM and has two 1GB connections to our SAN.

2. This weekend we plan to replace the server with 8 dual threaded cores with 32 GB RAM and give it two 4 GB connections to our SAN. (For those of you who understand that.) This will approximately double the processing power and throughput of our email system (it’s not linear)

3. Longer term, over the next two months, we will migrate email to a new configuration for email delivery involving six new servers where we currently have one.

4. Tues 17 Feb the IMAP server was rebooted. This eliminated many "orphaned" processes that would not normally matter but slowed the system down until they were cleaned up.

5. Sessions on IMAP are being controlled at manageable level.

6. We are script clearing old IMAP processes.

7. Spam is being controlled with more aggressive setting.

8. Monitoring, and system logs have been minimized on the IMAP server and this function has been moved to other servers.

9. Some of the University's mass emails are being postponed until the evening.

(The above was written by our Director. If you didn't understand some of that and want clarification on the details, please let me know and I will be happy to translate it in English.)

Thanks,

-Krista
In addition to the details above, I will provide some specific technical details of what is being done for those who want specifics:

1. Our current email server is a 4 dual threaded cores running at 2.7 Ghz with 16 MB RAM and has two 1GB connections to our SAN.

2. This weekend we plan to replace the server with 8 dual threaded cores with 32 GB RAM and give it two 4 GB connections to our SAN. (For those of you who understand that.) This will approximately double the processing power and throughput of our email system (it’s not linear)

3. Longer term, over the next two months, we will migrate email to a new configuration for email delivery involving six new servers where we currently have one.

4. Tues 17 Feb the IMAP server was rebooted. This eliminated many "orphaned" processes that would not normally matter but slowed the system down until they were cleaned up.

5. Sessions on IMAP are being controlled at manageable level.

6. We are script clearing old IMAP processes.

7. Spam is being controlled with more aggressive setting.

8. Monitoring, and system logs have been minimized on the IMAP server and this function has been moved to other servers.

9. Some of the University's mass emails are being postponed until the evening.

(The above was written by our Director. If you didn't understand some of that and want clarification on the details, please let me know and I will be happy to translate it in English.)

Thanks,

-Krista
Hi Krista.

I'm curious as to what the difference is between email in 2008 and email in 2009. From what I can recall everything started to slow down in mid-January, and it has gotten progressivly worse over the last four weeks. Is it the volume of email? Attachments? Spam? Did something on the server change? Blaming bulk emailing doesn't seem to make sense, as they never caused problems before.
Hi Krista.

I'm curious as to what the difference is between email in 2008 and email in 2009. From what I can recall everything started to slow down in mid-January, and it has gotten progressivly worse over the last four weeks. Is it the volume of email? Attachments? Spam? Did something on the server change? Blaming bulk emailing doesn't seem to make sense, as they never caused problems before.
Hello Mary,

We are not blaming mass emails. They are just one way among the others listed above that we are testing to see if it improves the situation.

We are not entirely sure what is causing the difficulties. We have been running tests on all of the systems. Email use has doubled since December but we have experienced these kinds of loads before, such as in September registration periods, and have not had this kind of breakdown. We do know that more people are connecting from wireless connections than ever before, and that Blackberrys appear to be about 25% of the usage, which is a huge increase.

Other than that, I can't give any more definite answers until our technicians have more information.

We will be keeping everyone updated as we get news. We are planning to give daily updates until this is resolved. I hope this means only two more updates. You will hear from me soon.

-Krista
I blame all the Facebook messages which people are getting sent to their Dal e-mails. Not large messages, but there's a lot of them. Just a thought for a contributing factor.
I blame all the Facebook messages which people are getting sent to their Dal e-mails. Not large messages, but there's a lot of them. Just a thought for a contributing factor.
Google Apps.

Dal mail has always been slow, cumbersome, and hard to manage. Not to mention we don't have nearly enough storage space. I've been migrating my dal emails to my Gmail account for the last year and it has worked fantastically. Dal might think about investing into Google Apps for their email.
I was really happy that when my outlook was crashing, I tried to log on through my.dal and there was a message there from Krista about the problems. It's just comforting to know that the problem is not something that I am doing - it is system-wide. Keep up the good work Krista, IT and the communications team.
Does this email problem effect incoming messages? If someone is sending me an email with an attachment, am I guaranteed to recieve it?
Does this email problem effect incoming messages? If someone is sending me an email with an attachment, am I guaranteed to recieve it?
Dal can't use Google Apps because of provincial legislation prohibiting the storage of personal information (including email) on servers located outside of Canada. It was passed in reaction to the US's PATRIOT Act. It's a piece of legislation that's causing IT departments (including the one I run) a great deal of problems.

As well, I don't know if you're aware of this, but Google's (and, by extension, GMail's) terms of service give Google the right to search through your email ad send you targeted advertising. I tend to recommend to people that they stay away from "free" webmail services as a result.
Dal can't use Google Apps because of provincial legislation prohibiting the storage of personal information (including email) on servers located outside of Canada. It was passed in reaction to the US's PATRIOT Act. It's a piece of legislation that's causing IT departments (including the one I run) a great deal of problems.

As well, I don't know if you're aware of this, but Google's (and, by extension, GMail's) terms of service give Google the right to search through your email ad send you targeted advertising. I tend to recommend to people that they stay away from "free" webmail services as a result.
I am concerned with the spam filtering increase, in the past I have had a crucial email from an employer end up in the junk email folder, but if the settings in the server are turned up does this mean I could end up never seeing some emails directed to me?
I am also concerned about spam filtering. In the past, I could pretty much be assured that everything in my spam box was junk, but in the past few months I've noticed quite a number of legitimate messages, including some from Dal colleagues with dal.ca addresses, end up in there.
I use webmail, and I have noticed if you log out with the log out button on the inbox screen you don't really log out. It's not until you log out on the main screen that you really log out. People who don't know this may think they're logged out when they arn't. I'm not a computer wizard or anything, but until I discovered this I used to get strange e-mails from myself to myself advertising things. At first I thought it was a virus, but the McAfee anti virus software dal offers didn't find anything when I scanned my system (I use a Mac too). Now I wonder if when I thought I was logged off (but not actually logged off) if advertisers were able to access my e-mail and use it. I wonder if this contributed to the problem -- especially if it happened to others?
I use webmail, and I have noticed if you log out with the log out button on the inbox screen you don't really log out. It's not until you log out on the main screen that you really log out. People who don't know this may think they're logged out when they arn't. I'm not a computer wizard or anything, but until I discovered this I used to get strange e-mails from myself to myself advertising things. At first I thought it was a virus, but the McAfee anti virus software dal offers didn't find anything when I scanned my system (I use a Mac too). Now I wonder if when I thought I was logged off (but not actually logged off) if advertisers were able to access my e-mail and use it. I wonder if this contributed to the problem -- especially if it happened to others?
OH MY GOD!!! I didn't realize Dal was having email problems!! Who would've thought?? Maybe you should send us an 20MB email with all the details, THANKS!

It's funny that we pay $8,000 a semester and can't get the features of a free email account available from Gmail or Hotmail. Most universities wouldn't have to mention this problem, they would realize it without needing to survey students and just FIX IT! That's clearly too much to ask. Please send more surveys about the issue, its definitely necessary.
Is the current server really only 4 dual threaded cores running at 2.7 Ghz with 16 MB RAM?
Heh, Facebook also searches through your personal info to target advertising, but I have a hunch that most Dal ITs have a facebook account. Not to mention the University itself.

It's unfortunate to hear about the provincial legislation, but Johnathan is right - with the amount of tuition we're paying there has got to be a better solution. I despise logging into WebCT/BLS as well, (prefers IE over FIREFOX??) which is necessary for most students who check their dal email accounts, and is always painfully slow and freezy. I haven't heard of a single student who likes using it.
Has Dalhousie investigated greylisting as a temporary alternative? (You can check it out on wikipedia). It would temporarily slow all legitimate email, but maybe not to the extent we're experiencing today.
Has Dalhousie investigated greylisting as a temporary alternative? (You can check it out on wikipedia). It would temporarily slow all legitimate email, but maybe not to the extent we're experiencing today.
I am a bit concerned with the announced more aggressive spam filtering.
Since Dal implemented a spam filter that intercepts messages "identified" as spam I lost several legitimate messages. As these messages are deleted and not stored in a spam folder I learned about this only by coincidence, i.e. people asked my why I did not respond to their message. As user I have no means to influence the settings of this initial spam filter. That this filter is highly efficient, sometimes a bit to aggressive, is pretty obvious by the small number of messages that end up in the spam folder controlled by user settings. Most of these are not even spam.
I hope more aggressive spam filtering in the future does not mean that the loss of legitimate messages will further increase, making communication by email a gamble. Given the fact that a lot of business communication is done by email and not just social chatting this would be highly undesirable.

Bottom line is, reading between the lines, that IT infrastructure improvements were long overdue to meet the increased demand on the Dal email system by its users.
I am a bit concerned with the announced more aggressive spam filtering.
Since Dal implemented a spam filter that intercepts messages "identified" as spam I lost several legitimate messages. As these messages are deleted and not stored in a spam folder I learned about this only by coincidence, i.e. people asked my why I did not respond to their message. As user I have no means to influence the settings of this initial spam filter. That this filter is highly efficient, sometimes a bit to aggressive, is pretty obvious by the small number of messages that end up in the spam folder controlled by user settings. Most of these are not even spam.
I hope more aggressive spam filtering in the future does not mean that the loss of legitimate messages will further increase, making communication by email a gamble. Given the fact that a lot of business communication is done by email and not just social chatting this would be highly undesirable.

Bottom line is, reading between the lines, that IT infrastructure improvements were long overdue to meet the increased demand on the Dal email system by its users.
Does anybody know who we can contact about the overkill spam filters? I get the idea they aren't reading this board, cause nobody has answered and I am still losing legitimate, potentially important emails. The spam filters have to be toned down, this is not an acceptable measure to take. Let the emails be slower until this is fixed but don't throw away MY mail cause the servers can't deliver it fast enough.
Does anybody know who we can contact about the overkill spam filters? I get the idea they aren't reading this board, cause nobody has answered and I am still losing legitimate, potentially important emails. The spam filters have to be toned down, this is not an acceptable measure to take. Let the emails be slower until this is fixed but don't throw away MY mail cause the servers can't deliver it fast enough.

Join the conversation, but keep it clean, stay on the topic and be brief. Read comments policy.

Characters remaining: 2000

(optional)