Berkeley Lab IT Status

(Completed) Tuesday, February 18, 2025, 6:00-7:00PM - Commons (Confluence) Maintenance

Commons was offline for approximately 50 minutes starting at 6:00 PM for an upgrade.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #upgrade #collaborationservices


(Resolved) Monday, February 17, 9:43 AM - Tuesday February 18, 10:20 AM, 2025 Commons (Confluence) Outage

10:20 AM - Resolved - Users can authenticate into Commons, regardless of their institution, without having to click "Allow me to pick from a list"

9:00 AM - Partially Resolved - Users can authenticate into Commons, but they will have to click on "Allow me to pick from a list" and then select the shibboleth option from the list.  During this time, access to Commons will be intermittent.

7:39 AM - The Collaboration Services Team is currently investigating an issue preventing users from authenticating into Commons.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #collaborationservices

(Completed) Sunday, February 16, 2025, 8:30 AM to 11:30 AM - Intermittent Telephone Service


Telephone Services will perform maintenance on the telephone system on Sunday, February 16, 2025 from 8:30am to 11:30am.  This maintenance may result in intermittent telephone service outages during the scheduled period.  Individuals are advised to dial 911 from a cellular phone if there is a life-threatening emergency or dial 486-6999 from a cellular phone to report incidents that are not life-threatening.


Please contact tsc@lbl.gov with questions or concerns regarding this outage.



(Resolved) Saturday, February 15, 2025, 1:31 PM - 7:59 PM Main LBL Hill Campus Power Event

Update Sat, Feb 15 8:15 pm

Power has been restored on site.  At this time we believe that all wired and wireless networks are functioning normally with the exception of the 180c/180d SSM construction trailers that remain offline.

Update Sat, Feb 15 4:20 pm

Facilities has confirmed a power event impacting many site buildings. Please see https://status.lbl.gov/ for general updates about the power event and power outage.

At this time we believe that buildings 180d, 37, 50, 55, 55a, 56, 56a, 56z, 60, 63, 64c, 65, 72, 72a, 72b, 72c, 74a, 74b, and 85b are completely without any wired or wireless networking or VoIP phones.

We believe that buildings 70 and 88 have portions of the buildings with wired and wireless network still available on generator or battery backup power and portions of the building without wireless or wired network.

We believe that all other LBL buildings have fully functional wired and wireless networking.  In buildings 2, 15, 59, 62, 64, 66, 67, 74, 83, 84, and 92 our network equipment is reporting that one redundant power supply has no power while the other one does have power, likely provided by generator or battery backup power.

We are continuing to coordinate with Facilities and SES to fully understand the electrical issues and monitor the IT systems.  We will work to restore communications services to buildings as soon as we can after power is restored. 

What’s Happening?

At 1:31 pm on Saturday, February 15, 2025 a power event occurred on the main LBL Hill Campus.

Many buildings on site lost all power, many others have only limited standby generator power, and many buildings are unaffected.  Central IT Division business systems and data centers were not impacted and remain fully functional.  LBLnet is investigating network and power status in each building and will provide updates when we have a more complete inventory of issues.


(Postponed) Thursday, January 23, 2025, 4:00 PM to 6:00 PM - BIG-IP Upgrade v15 to v17

Identity Management systems will be undergoing brief maintenance to upgrade load balancers. We do not expect any downtime of identity services. Visit this page for progress updates. Please contact the Identity Management Team (idm@lbl.gov) with questions or concerns regarding this maintenance.

#collaborationservices #idm #upgrade #maintenance

(Completed) Wednesday, January 22, 2025, 5:30-6:30PM - Commons (Confluence) Maintenance

Commons was offline for approximately 45 minutes starting at 5:30 PM for an upgrade.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #upgrade #collaborationservices

(Resolved) Tuesday, January 21, 2025, 7:30-10:00 AM - Bitbucket Outage

Bitbucket (https://bitbucket.org) was down due to a database issue which has been resolved.


More information about the Bitbucket outage is at:

https://bitbucket.status.atlassian.com/incidents/t5bcmddn4z26


#collaborationservices #outage #bitbucket

(Completed) Friday, January 10, 2025, 1:00 PM - Shibboleth Upgrade

Identity Management systems underwent maintenance to upgrade the service responsible for authentication at the Lab (Shibboleth).  


The maintenance lasted for 30 minutes and there does not appear to have been any outage for authentication services, including Single Sign-On, Multi-Factor Authentication, Phonebook, RADIUS (Employee Wireless and VPN), or Multifactor Authentication services.


For any questions or concerns regarding this maintenance, please contact the Identity Management Team at idm@lbl.gov.


#collaborationservices #idm #upgrade #maintenance

(Resolved) Monday, January 6, 2025 - Issues with login

Users attempting to login to Lab services may be presented with an error page.  The Identity Management Team is currently working on resolving the issue.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#idm #unplanned #outage


(Scheduled) Tuesday, December 24, 2024, to Wednesday, January 1, 2025, 8:00 AM to 8:00 PM each day

Site-wide Communications Maintenance Disruptions

What’s Happening?

The IT Division Infrastucture Services groups are planning many maintenance activities for the LBL communications systems including wired and wireless networks and telephones.  In order to minimize the impact of this work on LBL activities this work is scheduled to occur during the Winter Holiday break between 8:00 AM and 8:00 PM starting Tuesday, December 24 and running through Wednesday, January 1.

During these maintenance windows we expect that interruptions or outages may occur for up to 1 hour at a time, and any given location or system may experience more than one outage during this period.

We expect that the primary wired network in IT Division-managed data center spaces (50A-2109C, 50A-1275, and 50B-2275) and the wired networks provided by the ALS Controls and Beamline routers will not have outages for local, same-subnet communications, although they may have periods where some communication with remote subnets is disrupted.

On the afternoon of Wednesday, January 1 we plan to test a failure scenario where the main LBLnet networks have no access to ESnet and must rely completely on CENIC for external connectivity.  We expect that while routing tables on routers around the world are processing these updates that there might be short disruptions in the ability to talk to some remote networks.

Please contact lblnet@lbl.gov with questions about these wired and wireless interruptions.

Please contact tsc@lbl.gov with questions about these telephone interruptions.

(Completed) Sunday, December 8, 2024 3:00pm - 4:00pm Telephone Service Outage

Telephone Services will perform maintenance on the telephone system on 12/8/2024 from 3:00pm to 4:00pm.  Telephone services in buildings 88, 6, 50, 67,and 90 will be intermittent during that time.  Individuals are advised to dial 911 from a cellular phone if there is a life-threatening emergency or dial 486-6999 from a cellular phone to report incidents that are not life-threatening.

Please contact tsc@lbl.gov with questions or concerns regarding this outage.


Wednesday, December 4, 2024, 10:31 PM - Thursday, February 13, 2025 2:36 PM

Building 75 Complex Communications Outage

Update Thu, Feb 13 4:50 pm:  We have finally restored full network service to Building 75C.

Update Fri, Dec 6 11:15 am:  We have restored network and telephone services to Building 75B in addition to 75 and 75A from last night.  We do not have an ETA for restoring services to 75C yet.

Update Thu, Dec 5 8:00 pm:  We have restored network and telephone services to buildings 75 and 75A.  We expect to have services restored to 75B by mid-day tomorrow, Fri, Dec 6.  We do not have an ETA for restoring services to 75C yet.

Update Thu, Dec 5 2:00 pm:  We are hopefull that we will have services restored to the 75 complex by mid-day tomorrow, Fri, Dec 6.

What’s Happening?

All fiber optic communications to the Building 75 complex (including buildings 75, 75A, 75B, 75C, and 75D) was lost.  Wired and wireless network, mobile phone coverage, and desk phones are impacted.

We believe the outage is caused by rodent damage to underground cables.  Facilities and the IT Division are working on addressing the cause and restoring service.  We do not currently have an estimated time for partial or full restoration of services.

(Scheduled) Sunday, December 1, 2024, 8:00 PM to December 7, 2024, 12:00 OM 

CIFS File Shares on SPSS (Condo-Storage)

What’s Happening?

Facilities have scheduled an electrical power outage in Buildings 50B and 50F for infrastructure improvements. This will affect the CIFS Shares of SPSS V1 (condo-storage) system, commonly known as the G drive or S drive.

What You Need to Do

This outage is part of Facilities’ work to improve the Building 50B cooling system. 

We appreciate your cooperation and understanding during this time. If you have any questions or concerns, don’t hesitate to reach out to Kuldeep Chawla (kchawla@lbl.gov).

(Completed) Wednesday, November 27, 2024, 6:00AM-10:00AM - Intermittent Cellular Service in B90

Telephone Services will be doing maintenance on the B90 DAS cellular system on Wednesday 11/27/2024 from 6am-10am.  Cellular service inside B90 will be intermittent during this time.  If there is a life-threatening emergency, dial 911 from a Lab phone. WiFi will still be available during this maintenance. 

Please contact tsc@lbl.gov with questions or concerns regarding this outage.

(Completed) Wednesday, November 20, 2024, 6-6:30PM - Commons (Confluence) Maintenance

Commons was offline for approximately 30 minutes starting at 6 PM while Confluence was upgraded.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #upgrade #collaborationservices


(Completed) Saturday, November 16, 2024 9:00AM - 5:00pm Telephone Service Outage

Telephone Services performed maintenance on the telephone system on 11/16/2024 from 9:00am to 5:00pm.  Telephone services were intermittent during that time.  Individuals were advised to dial 911 from a cellular phone if there was a life-threatening emergency or dial 486-6999 from a cellular phone to report incidents that were not life threatening.

Please contact tsc@lbl.gov with questions or concerns regarding this outage.

(Completed) Saturday, November 9, 2024 16:00 - 20:37 UTC - Bitbucket Maintenance window

Atlassian Bitbucket underwent scheduled maintenance on Saturday, 11/9. During the maintenance window, all repositories in the berkeleylab workspace were Read-only. This scheduled maintenance affected Git via SSH, Git via HTTPS and Git LFS.

Please notify help@lbl.gov if you any issues with Bitbucket post-maintenance.

More details: https://bitbucket.status.atlassian.com/incidents/1kxwtmbqktls

#bitbucket #collab #maintenance

(Completed) Saturday, October 26th, 2024, 6:00AM - 10:00PM Telephone Service Outage in B978

PG&E interrupted electrical power to B978 (5885 Hollis Street in Emeryville) from 6am to 10pm on Saturday, October 26th, 2024. There was no access to the building during the power shutdown. Telephone service was not available in the building due to the power outage.

Please contact tsc@lbl.gov with questions or concerns regarding this outage.


(Resolved) Thursday, October 3, 2024 6:25AM - 7:51 AM - Commons Unscheduled Outage

Commons.lbl.gov (Confluence) experienced an unscheduled outage.  Service was re-established at ~7:50AM.

#commons #collab #outage

(Resolved) Monday, September 30 2024 - Verizon Wireless Outage

Verizon wireless experienced a major service outage starting at approximately 8am on 9/30/2024, affecting most of the USA, including their wireless services at LBNL. As of 12:15pm on 10/1/2024, Verizon reported service had been restored, although some users are still reporting service disruptions.


Please contact tsc@lbl.gov with questions or concerns regarding this outage.

(Completed) Friday, September 27 2024 - IdM Maintenance

Maintenance will be done to add redundancy to https://identity.lbl.gov/mfa

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#idm


(Resolved) Friday, September 6 2024, 8:45-9:05 AM - Issues with login

The issue with login.lbl.gov has been resolved.

For ~20 minutes, starting at ~8:45AM, users were unable to log into Lab services via SSO.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#idm


(Completed) Thursday, August 22, 2024, 6-6:45PM - Commons (Confluence) Maintenance

Commons was offline for approximately 30 minutes starting at 6PM while Confluence was upgraded.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #upgrade #collaborationservices


Tuesday, August 6, 2024 5:32PM - Wednesday, August 7, 2024 7:42AM - WiFi Authentication Outage

New session authentication attempts for the lbnl-employee and eduroam WiFi SSIDs were failing.  Other SSIDs (including lbnl-visitor and lbnl-visitor-offsite) were functioning normally.

Update Wednesday, August 7, 7:28AM:  The cause of the outage has been identified and the IT Division is working on restoring the service.  We hope that service will be restored soon.

Update Wednesday, August 7, 7:57AM:  We believe that service was restored at 7:42AM and that authentications are successful now.

#wifi #lblnet

(Resolved) Tuesday, August 6, 2024 5:51AM - Commons Unscheduled Outage

Commons.lbl.gov (Confluence) experienced an unscheduled outage which was resolved as of 8:25AM

#commons #collab

(Resolved) Friday August 2, 2024 8-8:25AM - Commons Unscheduled Outage

Commons.lbl.gov (Confluence) experienced an unscheduled outage which was resolved as of 8:25AM

#commons #collab

(Resolved) Tuesday July 30 10:45AM to 11:45 AM- Wireless Access Outage

Approximately 10% of Wireless Access Points were offline (~100) after the failure of a wireless network controller.  Access points have returned to service.  LBLnet is working with the vendor to understand the cause of the failure and develop a plan to avoid future outages.

#wifi #lblnet

(Completed) Wednesday, July 24, 2024, 5-6PM - RADIUS Maintenance

Services that use RADIUS may experience an outage as we upgrade the radiator software, although no outage is expected, for up to one hour starting at 5PM.

Please contact the IT Help Desk at idm@lbl.gov with questions or concerns.

#radius #upgrade #radiator #idm

(Completed) Thursday, July 18, 2024, 11:30PM Major Crowdstrike Issue 

Crowdstrike, the Labs antivirus provider, is causing Windows computer around the world to crash, with a blue screen of death. This is impacting Berkeley Lab, but to what extent is not clear at this time.  For more details please visit https://go.lbl.gov/crowdstrike-issue

Please contact the IT Help Desk at help@lbl.gov with questions or concerns. Phone and chat responses will be slower due to the extreme volume of calls.

#crowdstrike

(Completed) Wednesday, July 17, 2024, 5-5:30PM - Commons (Confluence) Maintenance

Commons was offline for approximately 20 minutes starting at 5PM while Confluence was upgraded.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #upgrade #collaborationservices

(Completed) Wednesday, July 10th, 2024 6:00PM - 9:00PM Telephone Services System Updates

LBLnet performed system updates on Wednesday, July 10th, from 6pm to 9pm. Some phone services may have been impacted during this window. 

Please contact tsc@lbl.gov with questions or concerns regarding this maintenance.

(Resolved) Thursday, June 27, 2024 - Password Reset Issues

IT is investigating reports from users experiencing issues resetting their Berkeley Lab passwords. 

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#password #idm

(Completed) Wednesday, June 26, 2024, 5:00-6:00PM - Conferences (Indico) Maintenance

Conferences was offline for approximately 1 hour starting at 5PM while the system was upgraded.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#conferences #upgrade #collaborationservices

(Completed) Wednesday, June 26, 2024, 6:00-7:00PM - Grouper Maintenance

Grouper (https://identity.lbl.gov/grouper) was offline for approximately 1 hour starting at 6PM while the system was updated.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#grouper #update #collaborationservices

(Completed) Friday, June 21, 2024, 5-6:30PM - Commons (Confluence) Maintenance

Commons was offline for approximately 1.5 hours starting at 5PM while the system was migrated.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #upgrade #collaborationservices

(Completed) Tuesday, June 18, 2024, 5-5:30PM - Commons (Confluence) Maintenance

Commons was offline for approximately 30 minutes starting at 5PM while Confluence was upgraded.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #upgrade #collaborationservices

(Completed) Thursday, June 6th, 2024 3:15PM - 6:15PM Avaya Workplace Soft Phone System Updates


Avaya Workplace Soft Phone had limited functionality starting at 3:15 PM through 6:15 PM Thursday, June 6th, 2024 for system updates.

Please contact Telephone Services (tsc@lbl.gov) with questions or concerns regarding this outage.


(Completed) Wednesday, June 5, 2024, 5-5:30PM - Commons (Confluence) Maintenance

Commons was offline for approximately 30 minutes starting at 5PM while the system was upgraded.

Please contact the IT Help Desk at help@lbl.gov with questions or concerns.

#commons #upgrade #collaborationservices