Berkeley Lab IT Status

(Completed) Saturday, March 23, 2024 8:00AM - 8:00PM Avaya Workplace Maintenance


Avaya Workplace Soft Phone had limited functionality starting at 8:00AM through 8:00PM, Saturday March 23, 2024 for system updates.

Please contact Telephone Services (tsc@lbl.gov) with questions or concerns regarding this outage.


(Completed) Thursday, March 21, 2024 6:00PM - IdM Maintenance

The Identity Management team upgraded the load balancers, which are responsible for routing most IdM traffic.

No outages are belived to have occurred.

Potentially impacted services include all operational authentication services.

Visit this page for progress updates.


Please contact the IT Help Desk (help@lbl.gov) with questions or concerns regarding this maintenance.


#collaborationservices #maintenance #idm

(Completed) Tuesday, March 19, 2024, 5-5:15PM - Commons (Confluence) Upgraded

Commons was offline for approximately 15 minutes starting at 5PM while the system was upgraded.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #upgrade

(Completed) Saturday, March 2, 2024 4:00PM - IdM Maintenance

The Identity Management team will be upgrading the load balancers responsible for routing most IdM traffic.

No outages are expected. Should an outage occur, it should last no longer than one hour.

Impacted service include all operational authentication services.

Visit this page for progress updates.


Please contact the IT Help Desk (help@lbl.gov) with questions or concerns regarding this maintenance.


#collaborationservices #maintenance #idm

(Completed) Tuesday, February 20, 2024, 5-5:30 PM - Commons was offline for an upgrade

commons.lbl.gov was offline for ~30 minutes from 5-5:30PM (PST) while Confluence was upgraded to the latest version. 

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #outage

(Completed) Tuesday, February 12-23, 2024 - IdM Maintenance

The Identity Management team will be migrating application routes from one load balancer to a new load balancer in order to upgrade the underlying EOL operating system with no significant down time.

Unnoticeable outages are expected. Visit this page for progress updates.

The following applications were known to be having issues on 2/16 AM, which should be fixed:

https://identity.lbl.gov/creategroups

https://identity.lbl.gov/apg 

Please contact the IT Help Desk (help@lbl.gov) with questions or concerns regarding this maintenance.


#collaborationservices #maintenance #idm

(Completed) Tuesday, January 30, 2024 - 7:00 AM to 9:00 AM- Doodle Maintenance

Doodle will be carrying out scheduled maintenance. This will last for approximately two hours - during which time Doodle may be slower than expected.


Visit this page for progress updates.

Please contact the IT Help Desk (help@lbl.gov) with questions or concerns regarding this maintenance.


#collaborationservices #maintenance #doodle

(Completed) Tuesday, January 30, 2024 - 2:00 PM - IdM Maintenance

Identity Management systems will be undergoing maintenance.  The maintenance is projected to last 30 minutes.

While most redundant applications are expected to operate normally, there is the possibility of a brief (on the order of minutes) outage.

These systems are two nodes running RADIUS services for HPCS. No other systems will be impacted.


Visit this page for progress updates.

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.


#collaborationservices #idm #maintenance #radius

(Completed) Wednesday, January 17, 2024 - 6:00 PM - IdM Maintenance

Identity Management systems will be undergoing maintenance.  The maintenance is projected to last 2 hours.

While most redundant applications are expected to operate normally, there is the possibility of a brief outage.

These systems include all the IT Division's authentication services, including Single Sign-On, Multifactor Authentication, LDAP, ancillary AD services, Phonebook, RADIUS (Employee Wireless and VPN), and Grouper. 


Visit this page for progress updates.

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.


#collaborationservices #idm #maintenance #bigips

(Completed) Tuesday, January 16, 2024, 10:30-10:50 AM - Commons was offline for an upgrade

commons.lbl.gov was offline for ~20 minutes starting at 10:30AM (PST) while Confluence was upgraded to latest, secure version. 

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #outage

(Completed) Thursday, January 11, 2024 - 4:00 PM - IdM Maintenance

https://password.lbl.gov will be reconfigured

The maintenance is projected to last one hour.

Availability will be intermittent during the maintenance window.

Visit this page for progress updates.

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.

#collaborationservices #idm #maintenance

(Completed) Tuesday, January 9, 2024 - 4:00 PM - IdM Maintenance

https://password.lbl.gov will be reconfigured

The maintenance is projected to last one hour.

Availability will be intermittent during the maintenance window.

Visit this page for progress updates.

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.

#collaborationservices #idm #maintenance

(Completed) Friday, January 5, 2024 - 3:00 PM - IdM Maintenance

The certificates for https://login.lbl.gov and https://phonebook.lbl.gov will be updated

The maintenance is projected to last one hour.

No outage is expected.

Visit this page for progress updates.

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.

#collaborationservices #idm #maintenance

(Completed) Saturday, December 23 and Tuesday, December 26, 2023 through Monday, January 1, 2024, 8:00 AM until 10:00 PM each day - LBLnet Winter Holiday Maintenance

Date: 23-DEC-2023 and 26-DEC-2023 through 01-JAN-2024
Time: 8:00 AM until 10:00 PM each day
Duration: 14 hours each day

Description: LBLnet is planning several maintenance tasks in almost all buildings over the holidays and curtailment period.  We will be installing updated software on many of our network devices. No IT-managed data center networks should be affected.  No Lab-wide production business systems nor the 50A-2109C colocation facility will be affected.

General Impact: On each of the days of the outage during the announced times, network devices will be receiving updated software, which may disrupt wired and wireless network services within buildings for periods of up to 1 hour. Other potentially disruptive network maintenance will be taking place on those days also. Some of our services will lose redundancy as we upgrade devices or change configurations, increasing the risk of a larger outage.  As we finalize the schedule of work we may send more detailed updates. If you are concerned about disruptions to your work, please contact us to discuss possible accommodations or a more detailed estimate of outages that might affect you.


Please contact lblnet@lbl.gov with questions or concerns regarding this work.  

Join the net-notify mailing list to receive regular updates about the Lab network.

#lblnet #outage 

(Resolved) Wednesday, December 27, 2023 10:00 AM until 12:05 PM - Telephone Service Outage
As a result of the scheduled LBLnet Winter Holiday Maintenance, telephone services were impacted on 12/27/23 at around 10:00am PT to buildings 2, 33, 46, 71, 76 and 977. 


LBLnet has restored network service to impacted buildings by 12:05pm PT and telephone services are back online.


Please contact tsc@lbl.gov with questions or concerns regarding this work.  

(Completed) Friday, December 22, 2023 - 3:00 PM - IdM Maintenance

Identity Management systems will be upgrading the Lab's directory service (LDAP).  

The maintenance is projected to last 2 hours.

While most redundant applications are expected to operate normally, there is the possibility of a brief outage.

These systems include LDAP, RADIUS (Employee Wireless and VPN), StartTLS, and MFA/linOTP


Visit this page for progress updates.

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.


#collaborationservices #idm #maintenance #ldap

(Completed) Wednesday, December 13, 2023, 5:00 PM - Conferences and PMP maintenance

conferences.lbl.gov and pmp.lbl.gov was offline for ~15 minutes on 12/13/23 starting at 5:00PM for scheduled maintenance.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#conferences #indico #pmp #outage

(Completed) Monday, December 11, 2023, 5:00 PM - Commons maintenance

commons.lbl.gov was offline for ~15 minutes on 12/11/23 starting at 5:00PM for scheduled maintenance.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #confluence #outage

(Completed) Wednesday, December 6, 2023, 9:40 AM - Commons has been upgraded

Commons was offline for ~20 minutes starting at 9:20AM while the upgrade was performed and is now accessible without authentication, unless pages and/or spaces are otherwise restricted.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #outage

(Completed) Thursday, November 9, 2023 - 8-8:30 PM - IdM / Wireless Maintenance

Identity Management systems related to Employee Wireless and eduroam (RADIUS applications) underwent maintenance. 

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.

#collaborationservices #idm #maintenance

(Resolved) Monday, November 6, 2023 - Issues with email delivery to DHS and other recipient addresses that use Proofpoint

Emails sent to dhs.gov (and other domains whose email security is managed by Proofpoint) addresses were bouncing due to an issues between Proofpoint and Service Now.

This has been resolved in a couple of ways:

1) A workaround, which or my or may not still be in place, was implemented by Proofpoint to allow messages from lbl.gov senders to be delivered to dhs.gov, doe.gov, dell.com, etc recipients

2) Emails from Service Now are being sent From @service-now.lbl.gov addresses and no longer come from an @lbl.gov sender: https://it.lbl.gov/changes-to-institutional-service-email-addresses/

Please contact emailteam@lbl.gov with questions or concerns regarding this issue.

(Completed) Tuesday, October 31, 2023, 10:15-10:45 AM - Commons upgraded

The Lab's Confluence application (commons.lbl.gov) was inaccessible from ~10:15-10:45AM while we upgraded to the latest version to fix a critical software vulnerability.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #outage

(Completed) Tuesday, October 23, 2023 - 6:00 PM - IdM Maintenance

MFA registration and bypass for new users as well as access to Grouper (https://identity.lbl.gov/grouper) was unavailable for about 1 hour starting at 6PM (PT) while Identity Management systems were undergoing maintenance.

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.

#collaborationservices #idm #maintenance

(Completed) Monday, October 23, 2023 - 6:00 PM - IdM Maintenance

MFA registration and bypass for new users as well as access to Grouper (https://identity.lbl.gov/grouper) was unavailable for about 1 hour starting at 6PM (PT) while Identity Management systems were undergoing maintenance.

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.

#collaborationservices #idm #maintenance

(Resolved) Saturday, October 21, 2023 - 1PM Phone Outage

An AT&T service outage at Leapfrog B971 impacted phone service.  AT&T has resolved their issue and phone service was restored at 2:04pm Tuesday, October 23, 2023.

Please contact Telephone Services (tsc@lbl.gov) with questions or concerns regarding this outage.

(Scheduled) Wednesday, October 23, 2023 - 5:30 PM - IdM Maintenance

Identity Management systems will be undergoing maintenance.  The maintenance is projected to last 2 hours.

While most redundant applications are expected to operate normally, there is the possibility of a brief outage.

These systems include all the IT Division's authentication services, including Single Sign-On, Multifactor Authentication, LDAP, ancillary AD services, Phonebook, RADIUS (Employee Wireless and VPN), and Grouper. 


Visit this page for progress updates.

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.


#collaborationservices #idm #maintenance

(Resolved) Wednesday October 11, 2023 ~ 3 AM - IdM unplanned outage

Users attempting to login were presented with following message:

Login failed. 

Your Berkeley Lab Identity account now requires the use of Multi-Factor Authentication (MFA).

You are required to enroll in MFA by following this linkbefore you can access any other Laboratory systems.

Note: You will be asked to login again when you access the MFA Management application.

This is due to an uplanned outage of the Multi-factor Authentication service.  The IdM Team is working on resolving the issue.

#outage #idm #mfa

(Completed) Wednesday, October 4, 2023, 5 PM - Commons maintenance

The Lab's Confluence application (commons.lbl.gov) was unavailable for about 30 minutes on Wednesday, October 4 starting at 5PM for scheduled maintenance.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #outage

(Resolved) Saturday, September 16, 2023 - 6:00 PM - Unscheduled MFA Outage

The nscheduled outage of login services, specifically when performing multifactor authentication, has been resolved.

#collaborationservices #idm #outage

(Completed) Wednesday, September 13, 2023 - 6:00 PM - IdM Maintenance

Identity Management systems will be undergoing maintenance.  The maintenance is projected to last 2 hours.

While most redundant applications are expected to operate normally, there is the possibility of a brief outage.

These systems include all the IT Division's authentication services, including Single Sign-On, Multifactor Authentication, LDAP, ancillary AD services, Phonebook, RADIUS (Employee Wireless and VPN), and Grouper. 


Visit this page for progress updates.

Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.


#collaborationservices #idm #maintenance

(Completed) Friday, September 8, 2023, 4:30 PM - Commons maintenance

The Lab's Confluence application (commons.lbl.gov) was unavailable for about 20 minutes on Friday, September 8 starting at 4:30PM for scheduled maintenance.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #outage

(Resolved) Thursday, September 7, 2023 - Issues with email delivery to DOE addresses

Emails sent to doe.gov addresses have bounced due to an issue with DOE email servers. The DOE IT team has restored mail delivery to their servers for emails originating from lbl.gov addresses. DOE IT is working on restoring email delivery from other domain addresses.

(Resolved) Tuesday, September 5, 2023, 12-1 PM - Commons

The Lab's Confluence application (commons.lbl.gov) experienced issues with InCommon users ability to log in, which were resolved at 1PM.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #outage

(Completed) Thursday, August 31 - Indico

conferences.lbl.gov (Indico) has been upgraded to a newer version.

Please contact indico-admins@lbl.gov with questions or concerns.

(Completed) Tuesday, August 22, 2023, 6-6:40 PM - Commons/Confluence

The Lab's Confluence application (commons.lbl.gov) has been upgraded to a newer version.  

Commons was offline for about 40 minutes starting at 6:00 PM.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#collaboration #commons #upgrade #confluence #atlassian

(Completed) Monday, August 14, 2023, 4:20 PM - Commons

The Lab's Confluence application (commons.lbl.gov) was migrated to the Data Center version.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#idm #phonebook #outage

(Resolved) Friday, August 11, 2023, 10:30 AM - Phonebook

The Phonebook application (phonebook.lbl.gov) experienced a brief outage this morning.  The service has been restored. 

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#idm #phonebook #outage

(Completed) Wednesday, August 2, 2023, 5:00 PM - Phonebook Self-Service

Phonebook Self-Service located at phonebook.lbl.gov will be upgraded starting at 5 PM.  The upgrade is expected to last 2 hours.  During this time, editing of your directory information (Telephone Numbers, Locations and Mailstop) will be disabled.  

For additional information on profiles at Berkeley Lab, visit go.lbl.gov/profileinformation.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#idm #hris #businesssystems #upgrade

(Completed) Sunday, July 30, 2023, 7:00 a.m. to 4:00 p.m. - Wireless Network Maintenance

Outage: LBLnet will be performing maintenance on redundant wireless controllers that may occasionally cause short periods of traffic loss.

General Impact: LBLnet is planning to upgrade our current wireless controllers.  No disruption to wireless connectivity is expected, but there is a risk that the ability for wireless clients may be interrupted in some areas if the controller swap does not complete successfully and the configuration needs to be backed out.  We expect that this interruption, if it occurs, would last under 30 minutes. 

Please contact LBLnet at lblnet@lbl.gov with questions or concerns regarding this outage.

(Resolved) Monday, July 24, 2023, 9:30AM 2:00PM - Intermittent AT&T Wireless In-Building Service

We were experiencing intermittent AT&T Wireless service within some buildings.  AT&T Wireless resolved their service issues and will continue to actively monitor the system for the next 24-hrs.  You may use WiFi Calling when a cellular carrier’s network is unavailable.

Please contact Telephone Services (tsc@lbl.gov) with questions or concerns.

(Resolved) Wednesday, July 5, 2023, 3:00 PM - Phonebook 

The Phonebook site (phonebook.lbl.gov) is currently not returning results when performing a search. The IdM team is currently investigating.  

You can use Google Contacts to look up employee information at contacts.google.com.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#idm #outage

(Completed) Wednesday, July 5, 2023, 5:30 p.m. to 6:00 p.m. - Lab Voice Mail Maintenance

The Lab voice mail system was unavailable Wednesday July 5, 2023 from 5:30pm - 6:00pm for planned maintenance. 

Please contact Telephone Services (tsc@lbl.gov) with questions or concerns regarding this maintenance.

Friday, June 30, 2023, 6:00pm - Required Update to Zoom Desktop Client v5.15.2

Issue: 

The IT Audio Visual Services team received a security notice regarding a vulnerability to the in-meeting chat feature in Zoom desktop client versions 5.15.0 and 5.15.1. Exposure of in-meeting chat messages intended to be encrypted by some Zoom clients may lead to disclosure of sensitive information. Refer to the Zoom Security Bulletin for more details. 

A minimum account-level desktop client requirement is enforced on the 3rd Thursday of each month. The current minimum client versions are 5.14.10 and 5.14.11 (last updated June 15, 2023).

Solution: 

AV Services will push an off-schedule update requirement to Zoom desktop client version 5.15.2 for all licensed users at 6:00 p.m. PDT on June 30, 2023. 

To install the latest version now, visit the Zoom Download Center.

Contact the IT Help Desk at help@lbl.gov for any related questions.

(Resolved) Friday, June 30, 2023, 8:34am - Lab Voice Mail Unavailable

The Lab voice mail system was unavailable from 7:00am until 8:34am.  Callers were unable to access messages or leave messages.  Services were restored at 8:34am.

Please contact Telephone Services (tsc@lbl.gov) with questions or concerns regarding this maintenance.

(Completed) Monday, June 26, 2023, 4:00 p.m. to 8:00 p.m. - IdM Maintenance

Outage: intermittent failures of services at the following services

https://identity.lbl.gov/IdentitySelfService2 

https://identity.lbl.gov/Shibboleth.sso/ 

https://identity.lbl.gov/OTP/TestHPCSEnrollment 

https://identity.lbl.gov/assets 

https://identity.lbl.gov/cgi-bin/ 

https://identity.lbl.gov/creategroups 

https://identity.lbl.gov/cyber-notices 

https://identity.lbl.gov/data 

https://identity.lbl.gov/identity-services/ 

https://identity.lbl.gov/linking 

https://identity.lbl.gov/linking/Shibboleth.sso 

https://identity.lbl.gov/maintenance 

https://identity.lbl.gov/mfa 

https://identity.lbl.gov/mfa/Shibboleth.sso 

https://identity.lbl.gov/mr 

https://identity.lbl.gov/news 

https://identity.lbl.gov/newsnotices 

https://identity.lbl.gov/otptokens

https://identity.lbl.gov/password/cgi-bin/ 

https://identity.lbl.gov/pulldata

General Impact: 

None expected. Contact idm@lbl.gov for questions or concerns.

(Resolved) Saturday, June 17, 2023, 7:00 a.m. to 4:00 p.m. - WiFi maintenance

All WiFi access points will be restarted to enable new features.  We expect that the interruption will last under 10 minutes per AP.  We expect wireless clients attached to each access point will be disconnected while this happens.  Most clients will reconnect automatically, but some may need to be reconnected by the user.

There was an issue in offsite buildings that caused us to roll back the changes.  We will attempt to complete those buildings in a future announced maintenance.

Please contact LBLnet at lblnet@lbl.gov with questions or concerns.

(Resolved) Wednesday June 14, 2023 4:55-5:15PM and 5:40-5:45pm - IdM unplanned outage

There was an unplanned outage of the Enterprise directory (ldaps://identity.lbl.gov:636) related to the new inCommon certificate which resolved

(Resolved) Wednesday, June 7, 2023, 8:45 p.m. to 10:30 p.m. - Gmail unplanned outage

An email configuration update to the Lab's Gmail service prevented inbound and outbound mail from being delivered to and being delivered from domains managed by the Lab between 8:45 pm and 10:30 pm last night.  This issue was resolved as soon as it was identified.  

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#collaborationservices #gmail #outage

(Resolved) Wednesday June 7, 2023 ~3 a.m. - 6:50 a.m. - IdM unplanned outage

There was an uplanned outage of login.lbl.gov with the following error message related to a change in the identity.lbl.gov certificate, which has been resolved.

Login failed with exception:

Pool is empty and connection creation failed

(Ongoing) Wednesday May 10, 2023Taleo Certificate Issue

IT is currently investigating an issue with Taleo.  This impacts recruitment, jobs.lbl.gov, and hiring/onboarding processes.   Workaround:  Visit https://lbl.referrals.selectminds.com/ to directly access Taleo.