• golden_zealot@lemmy.ml
    link
    fedilink
    English
    arrow-up
    22
    ·
    edit-2
    3 hours ago

    I used to perform data analysis of robotics firmware logs which would generate several million log lines per hour and that was my second job out of college.

    I don’t know how you fuck up 60k lines that bad. Is he nesting 150 for loops and loading a copy of the data set in each one while mining crypto??

    • ButtDrugs@lemm.ee
      link
      fedilink
      arrow-up
      6
      ·
      60 minutes ago

      Substring searches in unindexed large string columns or cartesian explosion caused by shitty joins would be my initial guess.