Citigroup Global Markets Inc. ("Citi") or "("CGMI") agreed to settle FINRA charges that it failed to retain emails - millions of them over a 14-month stretch - including emails pertinent to FINRA investigations.  The Relevant Period in this case ran from October 2008 to December 2009.  FINRA's accompanying charge relates to Citi's failure to have adequate systems and procedures in place for:  (i) achieving compliance with applicable recordkeeping rules; and, (ii) detecting and remedying deficiencies in its email retention systems. FINRA Findings and Allegations. In fall 2008, CGMI, thru parent company Citigroup Inc., began an upgrade of its email archiving system from a backup tape-based system to a new journaling-based system.  "Journaling," an email retention technology, electronically captures copies of all emails sent and received on a firm's mail servers and ultimately transmits those emails electronically to an email archive for storage.

Advantages of Journaling: For email archiving, it's faster, more secure and less error-prone than using backup tapes, provided the necessary servers are configured properly.  The journaling system was designed to function through the use of 3 component servers:  a mail server, a hub transport server, and a journaling server.

Under the journaling method, when an individual email user sent or received an email, the mail server sent a copy of that email to the hub transport server. The hub transport server then routed that email to the journaling server by utilizing a distribution list populated with the addresses of mailboxes on the journaling server. The journaled email was then electronically transferred by a 3rd-party vendor from the journaling server into the email archive.

Archiving Problems with Three Mail Servers. The upgraded email archiving system utilized 58 mail servers in North America.  With that large a number, Citi failed to detect that 3 of the 58 malfunctioned, and failed to properly transmit to archives any of the emails that were processed through those servers during parts or all of the Relevant Period. For one of the affected servers, CGMI, through Citigroup Inc., failed to re-activate the journaling function after populating the mail server with live email users.  Emails from this particular server therefore were not sent to the hub transport and journaling servers, and thus never reached the archive. For the other two affected servers, CGMI, through Citigroup Inc., failed to populate the distribution lists that the hub transport servers used to send emails to the journaling server with any mailbox addresses.  Thus, emails from these 2 servers never were sent to any destination for archiving. Emails for 2,800 Associated Persons. As a result, any and all incoming and outgoing emails for 2,800 associated persons that were processed through any of these 3 servers - covering up to 14 months - were lost.  Certain of the emails ultimately were retained by the Firm - e.g., when other CGMI personnel either sent or received those emails.  Recovery was impossible for those emails that were:  (i) exclusively between 2 or more CGMI persons whose emails were processed in an affected server;  (ii) were sent to or from other Citigroup Inc. employees on email servers outside North America whose emails were not being archived; and (iii) were sent or from individuals not associated with Citigroup Inc. Citi's Insufficient Testing and Quality Assurance Efforts. CGMI, through Citigroup, Inc., both before and after the system upgrade, failed to perform sufficient quality assurance ("QA") tests prior to migrating live users onto its upgraded email archiving system.  Initial stage QA testing was a manual process that had been insufficiently designed - e.g., it failed to detect the unpopulated distribution lists on the hub servers and the fact that the journaling function was never turned back on. An automated QA process was later developed that would have discovered issues, by neither CGMI nor Citigroup Inc. applied this automated QA process to each of its 58 mail servers, including the 3 Affected Servers.  Furthermore, after Citi, through Citigroup Inc., had completed the entire email archiving upgrade process for all 58 servers, the monitoring software that was used to check whether the journaling process was functioning properly was not sufficiently designed to flag reasonably foreseeable errors. It was not until the fall of 2009, when customized configuration management software that effectively communicated with a central Citigroup Inc. directory was developed, that CGMI, through Citigroup Inc., was able to flag that the 3 affected servers were problematic. FINRA Sanctions. Citigroup Global Markets, Inc., agreed to pay a $750,000 fine.  Citi received a reduced fine because it self-reported the email issues described in this case, and it conducted an internal review of its supervisory pols, procedures and systems relating to these issues. For further details, go to:   [FINRA AWC #20100218231]