All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

    • rozodru@lemmy.ca
      link
      fedilink
      English
      arrow-up
      79
      arrow-down
      2
      ·
      5 months ago

      yeah someone fucked up here. I mean I know you’re joking but I’ve been in tech for like 20+ years at this point and it was always, always, ALWAYS, drilled into me to never do updates on Friday, never roll anything out to production on Friday. Fridays were generally meant for code reviews, refactoring in test, work on personal projects, raid the company fridge for beer, play CS at the office, whatever just don’t push anything live or update anything.

      And especially now the work week has slimmed down where no one works on Friday anymore so you 100% don’t roll anything out, hell it’s getting to the point now where you just don’t roll anything out on a Thursday afternoon.

          • jedibob5@lemmy.world
            link
            fedilink
            English
            arrow-up
            2
            ·
            edit-2
            5 months ago

            Is the 4x10 really worth the extra day off? Tbh I’m not sure it would work very well for me… I find just one 10-hour day to be kinda draining, so doing that 4 times a week every week feels like it might just cancel out any benefits of the extra day off.

            • meanmon13@lemmy.zip
              link
              fedilink
              English
              arrow-up
              2
              ·
              5 months ago

              I am very used to it so I don’t find it draining. I tried 5x8 once and it felt more like working an extra day than getting more time in the afternoon. If that makes sense. I also start early around 7am, so I am only staying a little later than other people

        • rozodru@lemmy.ca
          link
          fedilink
          English
          arrow-up
          6
          ·
          5 months ago

          sorry :( yeah I, at most, do 3 days in the office now. Fridays are a day off and Mondays mostly everyone just works from home if at all. downtown Toronto on Mondays and Fridays is pretty much dead.

          • corsicanguppy@lemmy.ca
            link
            fedilink
            English
            arrow-up
            2
            ·
            5 months ago

            I changed jobs because the new management was all “if I can’t look at your ass you don’t work here” and I agreed.

            I now work remotely 100% and it’s in the union contract with the 21vacation days and 9x9 compressed time and regular raises. The view out my home office window is partially obscured by a floofy cat and we both like it that way.

            I’d work here until I die.

      • Blackmist@feddit.uk
        link
        fedilink
        English
        arrow-up
        15
        ·
        5 months ago

        Yep, anything done on Friday can enter the world on a Monday.

        I don’t really have any plans most weekends, but I sure as shit don’t plan on spending it fixing Friday’s fuckups.

        • sugar_in_your_tea@sh.itjust.works
          link
          fedilink
          English
          arrow-up
          3
          ·
          5 months ago

          And honestly, anything that can be done Monday is probably better done on Tuesday. Why start off your week by screwing stuff up?

          We have a team policy to never do externally facing updates on Fridays, and we generally avoid Mondays as well unless it’s urgent. Here’s roughly what each day is for:

          • Monday - urgent patches that were ready on Friday; everyone WFH
          • Tuesday - most releases; work in-office
          • Wed - fixing stuff we broke on Tuesday/planning the next release; work in-office
          • Thu - fixing stuff we broke on Tuesday, closing things out for the week; WFH
          • Fri - documentation, reviews, etc; WFH

          If things go sideways, we come in on Thu to straighten it out, but that almost never happens.

      • sasquash@sopuli.xyz
        link
        fedilink
        English
        arrow-up
        2
        ·
        5 months ago

        Actually I was not even joking. I also work in IT and have exactly the same opinion. Friday is for easy stuff!

    • merc@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      4
      ·
      5 months ago

      You posted this 14 hours ago, which would have made it 4:30 am in Austin, Texas where Cloudstrike is based. You may have felt the effect on Friday, but it’s extremely likely that the person who made the change did it late on a Thursday.

      • Toribor@corndog.social
        link
        fedilink
        English
        arrow-up
        58
        ·
        5 months ago

        This is fine as long as you politely ask everyone on the Internet to slow down and stop exploiting new vulnerabilities.

        • Ookami38@sh.itjust.works
          link
          fedilink
          English
          arrow-up
          22
          ·
          5 months ago

          I think vulnerabilities found count as “something broken” and chap you replied to simply did not think that far ahead hahah

          • huginn@feddit.it
            link
            fedilink
            English
            arrow-up
            14
            ·
            5 months ago

            For real - A cyber security company should basically always be pushing out updates.

            • sugar_in_your_tea@sh.itjust.works
              link
              fedilink
              English
              arrow-up
              4
              ·
              5 months ago

              Exactly. You don’t know what the vulnerabilities are, but the vendors pushing out updates typically do. So stay on top of updates to limit the attack surface.

              Major releases can wait, security updates should be pushed as soon as they can be proven to not break prod.

            • Midnight Wolf@lemmy.world
              link
              fedilink
              English
              arrow-up
              1
              ·
              5 months ago

              always pushing out updates

              Notes: Version bump: Eric is a twat so I removed his name from the listed coder team members on the about window.

              git push --force

              leans back in chair productive day, productive day indeed

          • sugar_in_your_tea@sh.itjust.works
            link
            fedilink
            English
            arrow-up
            2
            ·
            5 months ago

            I use Tumbleweed, so I only get updates once/day, twice if something explodes. I used to use Arch, so my update cycle has lengthened from 1-2x/day to 1-2x/week, which is so much better.

            • Nachorella@lemmy.sdf.org
              link
              fedilink
              English
              arrow-up
              2
              ·
              5 months ago

              I really like the tumbleweed method, seems like the best compromise between arch and debian style updates.

              • sugar_in_your_tea@sh.itjust.works
                link
                fedilink
                English
                arrow-up
                1
                ·
                edit-2
                5 months ago

                I think a lot of what (open)SUSE does is pretty solid. For example, microOS is a fantastic compromise between a stable base and a rolling userspace, and I think a lot of people would do well to switch to it from Leap. I currently use Leap for my NAS, but I do plan to switch to microOS.

      • Hotzilla@sopuli.xyz
        link
        fedilink
        English
        arrow-up
        2
        ·
        5 months ago

        This is AV, and even possible that it is part of definitions (for example some windows file deleted as false positive). You update those daily.