VITAC Services Interruption

On March 21, we discovered a network security incident in the VITAC environment. The following is meant to provide information about the impact, status, and measures being taken to remediate the situation.

favicon-Verbit-white (3)

Recent Updates

Updated – 12:00 PM EST on 4/5

  • VITAC is open for business and able to provide for your live and post-production captioning needs.
  • We appreciate your patience and support while we worked to bring our systems and services back online.

Updated – 12:20 PM EST on 3/31

  • Our live captioning department is in full operation and our availability in post-production continues to increase.
  • Live Captioning: We have secured all our services and our live captioning department is back in operation.
  • Post-Production: We’ve activated workflows and are now accepting short-form, quick-turnaround projects and transcription work in addition to English captions.

Updated – 5:50 PM EST on 3/29

  • Our order site (order.vitac.com) is up and ready to take orders.
  • We encourage order site users to reset their passwords. Simply follow the link on the order site page to receive a new password for your account.
  • We now are accepting Internet Captioning Service (ICS) and Zoom Integration orders via the website.

Updated – 11:00 AM EST on 3/29

  • We have restored operations but are not yet at full availability. Please contact your salesperson or email us for more details.
  • Live Captioning: VITAC Control Tower is operating and most captioners are working in the new secured environment.
  • Post-Production: We are in production but not yet at complete capacity. We continue to ask that customers verify their projects with their salesperson.

Updated – 6:05 PM EST on 3/27

  • We continue to bring more systems and offerings back online and available to customers.
  • Live Captioning: After securing all services, we have begun captioning via IP for some customers. Please bear with us while we build our availability.
  • Post-Production: As we continue to make more services available, we now are able to handle requests for multi-language service projects. We are not yet offering encoding services. Please contact your salesperson or email us for more details.

Updated – 4:35 PM EST on 3/26

  • We entered full testing mode yesterday. Our global task force is focusing on this around the clock, and we continue to see improvements to stability and availability in a new environment.
  • Live Captioning: We’ve stabilized delivery. Our order site and ICS remain down. We ask that live captioning (realtime) customers continue to verify schedules with order@vitac.com.
  • Post-Production: Our post-production captioning scheduler platform is now accessible internally and we are in production.

Updated – 1:00 PM EST on 3/25

  • We are still on track to restore operations early next week.
  • We completed the rebuild of our new environment and are continuing to test and harden security.
  • Live Captioning: We are beginning end-to-end testing of the new, secure environment.
  • Post-Production: We are receiving files from clients via SFTP (secure file transfer protocol) and we are in production.

Updated – 6:20 PM EST on 3/24

  • We have reconstructed our production environment and are in the process of testing and hardening to be ready for a gradual return to business as usual at the beginning of next week.
  • Live Captioning: We are working through the weekend using the current format for customers who have been using direct connections.
  • Post-Production: We now are accepting all video formats and sizes for existing customers via our new platform. Please contact offlinetraffic@vitac.com for more details.

Updated – 11:30 AM EST on 3/24

  • We are in the final stage of rebuilding and testing our new environment.
  • Live Captioning: We continue to ask that you verify schedules and connectivity by reaching out to order@vitac.com.
  • Post-Production: By the end of today, we will have three new, secure post-production upload solutions.

Updated – 6:35 PM EST on 3/23

  • We continue to progress with our external security analysis. We are making strides towards recovery.
  • Live Captioning: We successfully recovered much of our schedule from back-up. We ask that clients continue to verify with us their schedules and direct connection information by reaching out to order@vitac.com.
  • Post-Production: We can now accept captioning orders using MPG1 and WMV video formats delivered via links. We are not yet offering encoding services. Contact your salesperson for details.
  • We have been in active communication with the FCC. They are aware of the situation and understand that it may result in increased complaints. Please contact us for more information on how to deal with missing captions.

Updated – 2:30 PM EST on 3/23

  • The network security incident has been contained; we have identified the areas that have been impacted with a high degree of confidence.
  • We continue to make solid progress on recovering and rebuilding the network. The network has been set up in a new secure environment and is being tested and verified.
  • Post-Production: Though our post-production service delivery remains down, we have identified an alternative short-term solution for urgent needs. Please work with your sales representative to discuss your project.

Originally posted at 8:10 PM EST, March 22

We are in the process of recovering all systems and have shut down our VPN to prevent further unauthorized access. Our captioner stations have been verified as unaffected and safe. Our network in Canada has not been impacted nor has our ability to deliver captions via EEG iCap.

However, as of March 22, the following systems have been impacted:

We remain focused on minimizing the impact of this network security incident on our customers.

For live captioning customers: We have restored the ability to service your jobs though there remains the potential for interruption as we get back to full capacity. We anticipate that we will be able to restore full capacity over the coming days and weeks.

For ICS/Zoom customers using live elite: At this time, our ICS platform and Zoom integrations are unavailable and capacity very limited. If you require full coverage within the next 24 hours, we recommend looking for alternative solutions.

For clients using Evertz encoders: We can only connect via POTS audio and encoder connection and with limited capacity. Please forward your schedule and phone information to order@vitac.com.

For clients with EEG iCap encoders: We can safely provide this service, though at limited capacity. Please forward your schedule to order@vitac.com for the time being.

IP-enabled encoders without POTS backup: We continue to actively pursue a solution that is both workable and secure and will provide updates when we can resume activity.

For post-production customers: Our service delivery is currently down and we expect it to remain unavailable for the next 48-72 hours. We are actively exploring potential solutions that are workable and secure. There is a substantial likelihood that pending and uncompleted jobs will remain incomplete and may need to be re-submitted once we have a solution in place.

We will provide updates on this page as they become available.

If you have a specific inquiry about the status of your job, please contact your sales representative. If you do not know the name of your sales representative, email customer.questions@vitac.com.

Frequently Asked Questions

What happened?

We recently identified a network security incident that affected services for some of our VITAC customers.

What have you done about it?

Once we learned of the incident, we immediately deployed mitigation and containment actions, including shutting down the components that we believed could be potentially impacted, which resulted in a temporary outage of the scheduling system. We also shut down the VPN to block unauthorized access attempts. At the same time, we engaged an external forensics team to fully assess the nature of the incident, potential impact, and remediation options.

Have you reported this to the authorities?

Yes, we have reported this network security incident to the FBI.

Has customer data been exposed?

We are actively investigating whether, and to what extent, customer data has been impacted.

Why am I being asked to resend post-production files?

Given the nature of the incident, we proactively shut down all our servers, preventing us from having access to files that already have been shared. This does not mean that your files have been compromised.

Can this event spread to VITAC customers and partners?

Our top priority is to ensure that our customers’ systems and data remain safe. This is why we have taken immediate action to shut down the impacted elements, including our VPN.

Why did you wait to notify customers?

When our systems went down, it also limited our ability to identify those customers who were impacted.

Do you not have a backup plan?

The nature of the incident required us to shut down all our systems to make sure data was protected. We gradually are in the process of bringing those systems back up.

When will the incident be resolved?

We proactively and immediately deployed mitigation and containment actions, including shutting down the components that we believed could be potentially impacted. Our top priority is to protect our customers’ data and systems. We are devoting all efforts to investigating the matter fully, with the goal of securely restoring all services as soon as possible. At this time, we do not know when we will be able to restore all systems. We will update this page as we learn more and make progress.