Wells Fargo on Tealeaf
“Tealeaf is a one-of-a-kind solution that allows bank operators to identify problems immediately, reproduce them, rapidly isolate causal factors and prevent the problems from reoccurring. Errors that may have taken several days to diagnose can now be identified and resolved in a matter of hours.”-- Network Manager, Wells Fargo
Customer Profile - Wells Fargo is a diversified financial services company with $397 billion in assets, serving customers and partners from more than 5,900 stores and the Internet (wellsfargo.com). As one of the leading corporate internet sites, Wells Fargo has the resources, experience and technology necessary to help businesses succeed both online and off.
Vendor Profile - Tealeaf is the leading provider of online customer experience management solutions. Tealeaf's CX family of solutions provides unprecedented enterprise-wide visibility into every user's unique online interactions. This “360-degree view” of the online customer experience enables a clear and consistent understanding of the customer for ebusiness, IT, customer service and legal and compliance executives and their organizations across a wide range of vertical industries including retail, banking, travel, insurance, telecommunications, pharmaceutical and transportation. Founded in 1999, Tealeaf is headquartered in San Francisco, California, and is privately-held.
RETURN ON INVESTMENTS
• Significantly decreased IT support cost over first 6 months of use.
• Reduced problem resolution from several days to a few hours.
• Ensured high-value corporate clients enjoy same high-touch service online as offline.
• Boosted adoption of online service model through a more reliable web application.
“Simplifying your business life.”
This is exactly what the Commercial Electronic Office® (CEO®) Portal does for over 25,000 commercial banking customers, such as Chevron, Costco and Stanford University, that depend on Wells Fargo as a mission-critical banking resource. Providing single-sign-on to a personalized set of more than 40 wholesale banking services, Wells Fargo empowers its wholesale customers with a 24x7x365 mechanism to conduct critical, time-sensitive and high-value business transactions in a self-service fashion.
CHALLENGES OF ONLINE SELF SERVICE
The inability for customers to access online business services and successfully complete transactions, such as time-sensitive wire transfers, payment allocation or fraud-inquiry, not only complicates business operations for CEO portal customers, but exacts a significant business and IT cost to Wells Fargo.
Customer adoption and continual usage of the CEO portal is strategic both to providing faster service to customers anywhere, any time as well as to reduce the cost of all business transactions. Application failures, prolonged repair times and brute-force diagnosis not only result in a greater adverse business impact, but skyrocketing support costs and strained customer relationships. While Wells Fargo Wholesale Internet Solutions (WIS) had existing systems management and synthetic-based performance monitoring in place, WIS lacked real-time visibility into what was actually happening to the customer. As a result, WIS was unable to immediately reproduce the problem, rapidly isolate causal factors to mobilize the right IT resources, quantify rather than estimate true business impact, quickly identify the root cause to prevent further customer impact and proactively engage with the customer to maintain customer relationships.
A BETTER APPROACH
To solve this problem, Wells Fargo Wholesale Internet Solutions (WIS) turned to Tealeaf for a web application management solution that not only enabled the Wholesale Strategic Operations group within WIS to better manage the highly distributed, dynamic and constantly changing CEO environment, but better manage high-value customer relationships, meet internal audit and compliance requirements and deliver actionable metrics for business analysis.
REAL-TIME VISIBILITY—IMMEDIATE ROI
WIS selected Tealeaf because it is the only solution that passively provides outside-in real-time visibility into what 100 percent of the actual customers are doing and seeing at the browser level as they interact with online business services. Deployed across multiple data centers and used by a geographically dispersed support staff, Tealeaf provides immediate awareness into transaction-preventing problems that otherwise go undetected while enabling rapid resolution to avoid further impact. Requiring only a fraction of an FTE’s time to administer, Tealeaf has enabled the WIS team to:
• Improve overall customer success rate
• Significantly decrease IT support costs
• Immediately respond to impacted customers rather than wait days for customer impact reports to be created.
ISSUE NO. 1—FROM AWARENESS TO RESOLUTION IN TWO HOURS
Problem
A major release was migrated to production and during validation, problems were found such that most of the new code had to be backed out. The next day, Wells Fargo Treasury Management Customer Service reported a group of customers reporting intermittent errors while accessing the CEO Portal.
Solution
Using Tealeaf, a single WIS Strategic Operations Team member immediately reproduced the problem and assessed the business impact—over 80 impacted customers in a 24-hour period. Next, the problem was escalated to the appropriate application and system support team members. By reproducing the error and the recorded HTTP(S) request and response stream leading up to it, the support team immediately identified an orphaned servlet call requesting code that had been backed out. Having isolated the causal factors, systems support quickly blocked that call to prevent unnecessary ongoing impact. Then, using the same Tealeaf sessions, four additional orphaned servlet calls were identified—all before additional customer complaints were logged, thus preventing further support costs and strained customer relationships.
Benefit
Once escalated, the problem was resolved in under two hours. Without Tealeaf, significant additional, yet avoidable, impact and support costs would have been incurred.
ISSUE NO. 2—CUTTING RESOLUTION FROM TWO DAYS TO ONE HOUR
Problem
For two days, an application developer had been analyzing application server log files to debug an intermittent problem falsely diagnosed as linked to the login-functionality and user permissions. The problem impacted customers attempting to run account balance reports—a high-usage and time-sensitive CEO feature. Instead of seeing the requested report, users would receive an error message indicating there was no account data available. After growing customer complaints and little progress in resolving the issue, the problem was redirected to WIS and Tealeaf.
Solution
Using Tealeaf, WIS Strategic Operations immediately retrieved a recording of the actual customer session that included the error as well as one that didn’t. Using side-by-side analysis of the HTTP(S) request and response recording, Tealeaf revealed the intermittent report failures were correlated to an extra jsession call being submitted. Once isolated, the problem was quickly escalated to a third-tier support engineer who used the recorded Tealeaf Session to immediately identify the root cause—the default WebLogic application ID was being submitted instead of the actual defined application ID. A fix was implemented and the problem was resolved before it impacted additional customers.
Benefit
In one hour the problem was identified, reproduced and resolved—from the point of escalation to resolution. An impact assessment generated using Tealeaf indicated there was a 33 percent failure rate for customers during the days the problem was unresolved. Without turning to Tealeaf, failures could have continued indefinitely.
