Chase outage now at 48 hours

For Chase online banking customers the only news is the lack of news and specifically no new official statements (or any significant statements) from Chase as to the nature, severity, effects, and potential duration of the problem. For a blow-by-blow of today’s events, see our earlier post.

In the absence of any official statements by Chase, we have had to rely on unsubstantiated reports that we’ve seen in our comments and through Twitter.  One such tweet recently suggested a fairly severe problem:

Chase outage is a result of corruption to the database clusters containing customer online profiles (ie bil pay settings, usernames etc.).

Depending on the reliability of Chase’s backup system, this could result in some pretty inconvenient loss of data for customers.

The outage, which by our best estimates started sometime between 7-10 pm ET on Monday, and has included online banking being completely disabled/unavailable from Monday evening until early Wednesday morning, and sporadic and intermittent access since Wednesday morning, has now exceeded 48 hours and is believed by many to be the longest duration online banking outage for a bank of Chase’s size.

Chase online banking still experiencing problems

We’ve received quite a few comments (starting at around 6:40 am PT/9:40 am ET) that Chase’s website is still experiencing problems.  The most common experience seems to be the site timing out after someone tries to log in.  Some people were able to log in earlier but are having problems now.  One reader reports his login being rejected despite entering the correct user name and password.

One customer received the following error, clearly from the servers behind the scenes:

This error code is native to internal exceptions from the Oracle programs. It indicates that the process has encountered an unexpected low. It may be due to
•
•timeout
•A corrupted file
•crash verification data in memory
•problem of I/O of physical memory

Perhaps this is why they haven’t offered an official explanation … they haven’t really solved it yet.

Update 8:20 am PT/11:20 am ET:  Its officially down again.

As a side note, when I was looking at the website source for the main page this morning while the site was still up, I noticed that the language for the site being down was still in there, just not active.  This might be a sign that Chase has known since they put the site up again last night that they didn’t fix the problem.

Update 9:00 am PT/12 noon ET:  The login screen is active again.  I think it is safe to assume you can’t count on the site for the time being.

At this point (and well before this point) Chase has completely failed in its responsibility to get information out to its customers.

Update 9:30 am PT/12:30 pm ET:  Still receiving reports of log in failures.  When I try to load the main page (www.chase.com) it never quite finishes loading

Update 11:30 PT/2:30 ET:  Still receiving reports of inability to log in.  Does Chase think that having the login screen present despite widespread inability to log in is less of a black eye than having an system down message on their site?  The media is reporting Chase online banking as problematic regardless.

Update 1:30 pm PT/4:30 pm ET:  Still receiving reports of inability to log in.  Not fixed yet.

Update 3:30 pm PT/6:30 pm ET:

• Problems persist.
• Chase is acknowledging that there are still problems (gee thanks).
• I’ve seen several reports that this is now the longest outage ever of a bank this size.
• Chase has no comment on why it is not communicating with customers what is going on.

Update 4:30 pm PT/7:30 pm ET:

Still lots of reports of the same problems.  No new information of any kind from Chase.  Their lack of public statements about this problem is disappointing, and this far into it very irresponsible.

I’ve seen several comments from people claiming that they are seeing unauthorized activity on their accounts, in the brief moments when they can actually see their balances.  Another reader contacted me with this story:

I was able to log into my account this morning, but that balance was all wrong. I had double charges, and the calcuations didn’t add up right. I have NO IDEA how much I have in the bank right now. I’m glad I opened a new account a few days ago at a different bank. I’m just waiting for Chase to get their shit together so I can close the account. I feel like a hostage.

I don’t think there is enough information to make a significant generalization from these comments, but the potential trend is worrisome enough to mention.

Update 5:30 pm PT/8:30 pm ET:  Just saw this tweet:

Chase outage is a result of corruption to the database clusters containing customer online profiles (ie bil pay settings, usernames etc.).

This is completely hearsay, but without Chase providing any official statements to clue people in, this is what we have to go with.  It is surely a coincidence that I found quite a few listings for open database administrator jobs JP Morgan Chase has posted.

Chase online banking outage – 10 pm PT/1 am ET update – IT’S UP!

Our test server logs show that at 10:03 pm PT/1:03 am ET the chase.com server abruptly closed a connection to our test program and this is presumably when they brought the site back up.  It is now 10:30 pm PT/1:30 am ET and the chase.com main page is once again showing the login fields.  Ok so they squeaked by with their statement that the service would be back up “this evening”.

That puts this outage at around 28-30 hours long.

Since I don’t personally have a Chase account, I can’t try to log in and check the online banking and credit card services.  If you are a Chase customer and have any kind of experience using the website now, please leave word of what you find in the comments.

Chase online banking down – 7:45 pm PT/10:45 pm ET update

According to this article in the Wall Street Journal, Chase is just now clarifying the nature of the problem and the time frame for online banking to come back up:

A spokesman for the nation’s second-largest bank by assets said the service went down because of a “technical issue, not hackers or criminals.” Another person familiar with the situation said the disruption was the result of a flaw in a software program tailored for J.P. Morgan.

It took the bank much of Tuesday to discover the flaw and figure out how to fix it across the system, this person said. The bank expected the online service, which affected computer and iPhone users, to be fully restored Tuesday evening, this person added.

It is now almost 11 pm on the east coast, so they are straining the definition of evening, but perhaps they will claim west coast or Hawaii time to justify this statement.  We’ll see if it comes back up before Tuesday rolls to Wednesday here on the west coast.

They still have not provided any more detailed information on their website, the most obvious and efficient place to distribute up to date information on this problem.

Chase online down – 4 pm PT/7 pm ET update

More comments from Chase’s spokesman, Tom Kelly:

The bank will work with any customers who had online bill payment deadlines they couldn’t meet because of the outage, Kelly said.

We’ve had one customer report in the comments that an incoming ACH transfer was successful, but according to Kelly’s statements, we can assume there are problems with online bill payments.  It might be a good idea to contact your payees or make other arrangements (like a good old fashioned check in the mail).

Again in summary from our 1 pm PT/4 pm ET update, branches, ATM’s and telephone banking are operational although telephone banking still has a considerable wait.

In another statement Tom Kelly says:

The cause of the outage isn’t clear, Kelly said. “It’s a technical issue that we’re dealing with.”

So, clearly they won’t be able to give any estimate on when the online banking might be back up.

If anyone is interested, this is the contact information I found for Chase’s spokesman:

Thomas Kelly
Chase Media Contact
(312) 732 – 7007
cell: (312) 848-5265
thomas.a.kelly@chase.com

Chase online down – 1 pm PT/4 pm ET update

Piecing together various reports, we are pegging the start of this outage was between 4 and 6 pm PT (7 and 9 pm ET) yesterday.  That means the outage is now 19-21 hours old. It is likely that this outage was preceded by periodic problems for as many as 48 hours before the full-scale outage started.

Chase ran “down due to scheduled maintenance” message until 12:30 am PT/3:30 am ET this morning and then changed to a “temporarily unavailable” message which they are still running.

So far we’ve only seen two reports of statements by Chase, one by spokesman Tom Kelly:

The problem began overnight, and the company is working to fix it

bank spokesman Thomas Kelly couldn’t specify the exact time service was disrupted.

This statement appears to be somewhat misleading as many reports put the start of the outage earlier than “overnight.”  In another report:

Bank spokesman Thomas Kelly couldn’t specify the exact time service was disrupted.

It seems possible that Chase may not have a good handle on when the problem started.

In another comment CEO Jamie Dimon apologized for the outage.  Chase has made no other official statements about the cause or duration of the outage that we have seen.

We’ve seen various reports from people who say they have talked to customer service reps that claimed the site would be back up as soon as 30 minutes (at 10 am PT/1 pm ET) which it obviously hasn’t, to 3 hours (11:30 PT/2:30 ET).  I think it is safe to say that Chase may themselves not have a good idea of how long this problem will take to resolve.

Reports seem to indicate that branch operations, ATM’s and telephone banking are all functioning properly, although phone banking has a considerable wait.  By some reports branch traffic is light.  There have been no reports on whether online transactions such as money transfers or bill payments are functioning.

That is all the information we’ve seen so far.

Chase online down – where is the media?

You would think that a 24 hours outage on the second largest bank in the US would elicit a bit more of a response from the mainstream media.  So far, I the only large media company that appears to have any information on the Chase bank online outage is Bloomberg.

Included in the sites I have checked for any trace of information (but found none) are the Wall Street Journal, New York Times, LA Times, Fox Business, San Francisco Chronicle, and USA Today.  Checking Google News ought to turn up stories on ANY mainstream media site, but there is nothing.

Update:  Now Fox Business is running the story, but it is the same thing Chicago Breaking Business reported hours ago.

Update:  Now ABC News is running the story.

Update 1 pm PT/4 pm ET:  Ok, now the mainstream media is getting on board.  The New York Times, LA Times, and San Francisco Chronicle all are repeating essentially the same story we’ve seen earlier on Chicago Breaking Business and Bloomberg earlier today.  So far no one is reporting any more details probably because Chase hasn’t released any.

Chase online experiencing problems for three days?

I Just saw a tweet claiming Chase’s website has been down for three days.  While I don’t think it was down for everyone for three days, this might be an indication that they were having sporadic problems days before yesterday’s outage started.

Chase claims, through a spokesman, that their branch and phone banking is operational, although phone banking is experiencing a very long wait.  It is unfortunate that they haven’t provided more detailed information on the nature and possible duration of the problems so people can make plans about transactions that might be affected.