Berkeley Lab IT Status

(Completed) Thursday, November 9, 2023 - 8-8:30 PM - IdM / Wireless Maintenance

Identity Management systems related to Employee Wireless and eduroam (RADIUS applications) underwent maintenance. 

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.

#collaborationservices #idm #maintenance

(Resolved) Monday, November 6, 2023 - Issues with email delivery to DHS and other recipient addresses that use Proofpoint

Emails sent to dhs.gov (and other domains whose email security is managed by Proofpoint) addresses were bouncing due to an issues between Proofpoint and Service Now.

This has been resolved in a couple of ways:

1) A workaround, which or my or may not still be in place, was implemented by Proofpoint to allow messages from lbl.gov senders to be delivered to dhs.gov, doe.gov, dell.com, etc recipients

2) Emails from Service Now are being sent From @service-now.lbl.gov addresses and no longer come from an @lbl.gov sender: https://it.lbl.gov/changes-to-institutional-service-email-addresses/

Please contact emailteam@lbl.gov with questions or concerns regarding this issue.

(Completed) Tuesday, October 31, 2023, 10:15-10:45 AM - Commons upgraded

The Lab's Confluence application (commons.lbl.gov) was inaccessible from ~10:15-10:45AM while we upgraded to the latest version to fix a critical software vulnerability.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #outage

(Completed) Tuesday, October 23, 2023 - 6:00 PM - IdM Maintenance

MFA registration and bypass for new users as well as access to Grouper (https://identity.lbl.gov/grouper) was unavailable for about 1 hour starting at 6PM (PT) while Identity Management systems were undergoing maintenance.

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.

#collaborationservices #idm #maintenance

(Completed) Monday, October 23, 2023 - 6:00 PM - IdM Maintenance

MFA registration and bypass for new users as well as access to Grouper (https://identity.lbl.gov/grouper) was unavailable for about 1 hour starting at 6PM (PT) while Identity Management systems were undergoing maintenance.

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.

#collaborationservices #idm #maintenance

(Resolved) Saturday, October 21, 2023 - 1PM Phone Outage

An AT&T service outage at Leapfrog B971 impacted phone service.  AT&T has resolved their issue and phone service was restored at 2:04pm Tuesday, October 23, 2023.

Please contact Telephone Services (tsc@lbl.gov) with questions or concerns regarding this outage.

(Scheduled) Wednesday, October 23, 2023 - 5:30 PM - IdM Maintenance

Identity Management systems will be undergoing maintenance.  The maintenance is projected to last 2 hours.

While most redundant applications are expected to operate normally, there is the possibility of a brief outage.

These systems include all the IT Division's authentication services, including Single Sign-On, Multifactor Authentication, LDAP, ancillary AD services, Phonebook, RADIUS (Employee Wireless and VPN), and Grouper. 

Visit this page for progress updates.

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.

#collaborationservices #idm #maintenance

(Completed) Wednesday, October 4, 2023, 5 PM - Commons maintenance

The Lab's Confluence application (commons.lbl.gov) was unavailable for about 30 minutes on Wednesday, October 4 starting at 5PM for scheduled maintenance.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #outage

(Resolved) Saturday, September 16, 2023 - 6:00 PM - Unscheduled MFA Outage

The nscheduled outage of login services, specifically when performing multifactor authentication, has been resolved.

#collaborationservices #idm #outage

(Completed) Wednesday, September 13, 2023 - 6:00 PM - IdM Maintenance

Identity Management systems will be undergoing maintenance.  The maintenance is projected to last 2 hours.

While most redundant applications are expected to operate normally, there is the possibility of a brief outage.

These systems include all the IT Division's authentication services, including Single Sign-On, Multifactor Authentication, LDAP, ancillary AD services, Phonebook, RADIUS (Employee Wireless and VPN), and Grouper. 

Visit this page for progress updates.

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.

#collaborationservices #idm #maintenance

(Completed) Friday, September 8, 2023, 4:30 PM - Commons maintenance

The Lab's Confluence application (commons.lbl.gov) was unavailable for about 20 minutes on Friday, September 8 starting at 4:30PM for scheduled maintenance.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #outage

(Resolved) Thursday, September 7, 2023 - Issues with email delivery to DOE addresses

Emails sent to doe.gov addresses have bounced due to an issue with DOE email servers. The DOE IT team has restored mail delivery to their servers for emails originating from lbl.gov addresses. DOE IT is working on restoring email delivery from other domain addresses.

(Resolved) Tuesday, September 5, 2023, 12-1 PM - Commons

The Lab's Confluence application (commons.lbl.gov) experienced issues with InCommon users ability to log in, which were resolved at 1PM.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #outage

(Completed) Thursday, August 31 - Indico

conferences.lbl.gov (Indico) has been upgraded to a newer version.

Please contact indico-admins@lbl.gov with questions or concerns.

(Completed) Tuesday, August 22, 2023, 6-6:40 PM - Commons/Confluence

The Lab's Confluence application (commons.lbl.gov) has been upgraded to a newer version.  

Commons was offline for about 40 minutes starting at 6:00 PM.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#collaboration #commons #upgrade #confluence #atlassian

(Completed) Monday, August 14, 2023, 4:20 PM - Commons

The Lab's Confluence application (commons.lbl.gov) was migrated to the Data Center version.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#idm #phonebook #outage

(Resolved) Friday, August 11, 2023, 10:30 AM - Phonebook

The Phonebook application (phonebook.lbl.gov) experienced a brief outage this morning.  The service has been restored. 

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#idm #phonebook #outage

(Completed) Wednesday, August 2, 2023, 5:00 PM - Phonebook Self-Service

Phonebook Self-Service located at phonebook.lbl.gov will be upgraded starting at 5 PM.  The upgrade is expected to last 2 hours.  During this time, editing of your directory information (Telephone Numbers, Locations and Mailstop) will be disabled.  

For additional information on profiles at Berkeley Lab, visit go.lbl.gov/profileinformation.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#idm #hris #businesssystems #upgrade

(Completed) Sunday, July 30, 2023, 7:00 a.m. to 4:00 p.m. - Wireless Network Maintenance

Outage: LBLnet will be performing maintenance on redundant wireless controllers that may occasionally cause short periods of traffic loss.

General Impact: LBLnet is planning to upgrade our current wireless controllers.  No disruption to wireless connectivity is expected, but there is a risk that the ability for wireless clients may be interrupted in some areas if the controller swap does not complete successfully and the configuration needs to be backed out.  We expect that this interruption, if it occurs, would last under 30 minutes. 

Please contact LBLnet at lblnet@lbl.gov with questions or concerns regarding this outage.

(Resolved) Monday, July 24, 2023, 9:30AM 2:00PM - Intermittent AT&T Wireless In-Building Service

We were experiencing intermittent AT&T Wireless service within some buildings.  AT&T Wireless resolved their service issues and will continue to actively monitor the system for the next 24-hrs.  You may use WiFi Calling when a cellular carrier’s network is unavailable.

Please contact Telephone Services (tsc@lbl.gov) with questions or concerns.

(Resolved) Wednesday, July 5, 2023, 3:00 PM - Phonebook 

The Phonebook site (phonebook.lbl.gov) is currently not returning results when performing a search. The IdM team is currently investigating.  

You can use Google Contacts to look up employee information at contacts.google.com.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#idm #outage

(Completed) Wednesday, July 5, 2023, 5:30 p.m. to 6:00 p.m. - Lab Voice Mail Maintenance

The Lab voice mail system was unavailable Wednesday July 5, 2023 from 5:30pm - 6:00pm for planned maintenance. 

Please contact Telephone Services (tsc@lbl.gov) with questions or concerns regarding this maintenance.

Friday, June 30, 2023, 6:00pm - Required Update to Zoom Desktop Client v5.15.2

Issue: 

The IT Audio Visual Services team received a security notice regarding a vulnerability to the in-meeting chat feature in Zoom desktop client versions 5.15.0 and 5.15.1. Exposure of in-meeting chat messages intended to be encrypted by some Zoom clients may lead to disclosure of sensitive information. Refer to the Zoom Security Bulletin for more details. 

A minimum account-level desktop client requirement is enforced on the 3rd Thursday of each month. The current minimum client versions are 5.14.10 and 5.14.11 (last updated June 15, 2023).

Solution: 

AV Services will push an off-schedule update requirement to Zoom desktop client version 5.15.2 for all licensed users at 6:00 p.m. PDT on June 30, 2023. 

To install the latest version now, visit the Zoom Download Center.

Contact the IT Help Desk at help@lbl.gov for any related questions.

(Resolved) Friday, June 30, 2023, 8:34am - Lab Voice Mail Unavailable

The Lab voice mail system was unavailable from 7:00am until 8:34am.  Callers were unable to access messages or leave messages.  Services were restored at 8:34am.

Please contact Telephone Services (tsc@lbl.gov) with questions or concerns regarding this maintenance.

(Completed) Monday, June 26, 2023, 4:00 p.m. to 8:00 p.m. - IdM Maintenance

Outage: intermittent failures of services at the following services

https://identity.lbl.gov/IdentitySelfService2 

https://identity.lbl.gov/Shibboleth.sso/ 

https://identity.lbl.gov/OTP/TestHPCSEnrollment 

https://identity.lbl.gov/assets 

https://identity.lbl.gov/cgi-bin/ 

https://identity.lbl.gov/creategroups 

https://identity.lbl.gov/cyber-notices 

https://identity.lbl.gov/data 

https://identity.lbl.gov/identity-services/ 

https://identity.lbl.gov/linking 

https://identity.lbl.gov/linking/Shibboleth.sso 

https://identity.lbl.gov/maintenance 

https://identity.lbl.gov/mfa 

https://identity.lbl.gov/mfa/Shibboleth.sso 

https://identity.lbl.gov/mr 

https://identity.lbl.gov/news 

https://identity.lbl.gov/newsnotices 

https://identity.lbl.gov/otptokens

https://identity.lbl.gov/password/cgi-bin/ 

https://identity.lbl.gov/pulldata

General Impact: 

None expected. Contact idm@lbl.gov for questions or concerns.

(Resolved) Saturday, June 17, 2023, 7:00 a.m. to 4:00 p.m. - WiFi maintenance

All WiFi access points will be restarted to enable new features.  We expect that the interruption will last under 10 minutes per AP.  We expect wireless clients attached to each access point will be disconnected while this happens.  Most clients will reconnect automatically, but some may need to be reconnected by the user.

There was an issue in offsite buildings that caused us to roll back the changes.  We will attempt to complete those buildings in a future announced maintenance.

Please contact LBLnet at lblnet@lbl.gov with questions or concerns.

(Resolved) Wednesday June 14, 2023 4:55-5:15PM and 5:40-5:45pm - IdM unplanned outage

There was an unplanned outage of the Enterprise directory (ldaps://identity.lbl.gov:636) related to the new inCommon certificate which resolved

(Resolved) Wednesday, June 7, 2023, 8:45 p.m. to 10:30 p.m. - Gmail unplanned outage

An email configuration update to the Lab's Gmail service prevented inbound and outbound mail from being delivered to and being delivered from domains managed by the Lab between 8:45 pm and 10:30 pm last night.  This issue was resolved as soon as it was identified.  

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#collaborationservices #gmail #outage

(Ongoing) Wednesday May 10, 2023 -  Taleo Certificate Issue

IT is currently investigating an issue with Taleo.  This impacts recruitment, jobs.lbl.gov, and hiring/onboarding processes.   Workaround:  Visit https://lbl.referrals.selectminds.com/ to directly access Taleo.