1 00:00:00,480 --> 00:00:03,480 foreign 2 00:00:09,980 --> 00:00:14,580 thank you all very much for joining us 3 00:00:12,420 --> 00:00:16,560 again we have our third speaker in this 4 00:00:14,580 --> 00:00:19,020 block Courtney Eckert who will be 5 00:00:16,560 --> 00:00:20,640 speaking to us about uh or giving us a 6 00:00:19,020 --> 00:00:21,900 crash course in having an operational 7 00:00:20,640 --> 00:00:23,580 incident something that will never 8 00:00:21,900 --> 00:00:25,320 happen with the good Folks at uh next 9 00:00:23,580 --> 00:00:28,400 day video running the show 10 00:00:25,320 --> 00:00:28,400 take it away Courtney 11 00:00:30,240 --> 00:00:33,480 good morning or good afternoon or good 12 00:00:32,220 --> 00:00:36,059 evening depending on where in the world 13 00:00:33,480 --> 00:00:38,040 you are I would like to begin with some 14 00:00:36,059 --> 00:00:39,719 content notes I will be discussing a 15 00:00:38,040 --> 00:00:41,399 number of upsetting topics in theory 16 00:00:39,719 --> 00:00:43,800 because I'm drawing an allergy to 17 00:00:41,399 --> 00:00:46,140 Emergency Response situations in the 18 00:00:43,800 --> 00:00:48,300 physical world there will be no details 19 00:00:46,140 --> 00:00:50,039 and no actual scenarios I won't be going 20 00:00:48,300 --> 00:00:51,600 into any of this until slide eight so if 21 00:00:50,039 --> 00:00:53,160 you need to step out you have time to do 22 00:00:51,600 --> 00:00:55,440 so please do what you need to do to take 23 00:00:53,160 --> 00:00:58,440 care of yourself content notes are 24 00:00:55,440 --> 00:01:00,660 theoretical discussions of fires natural 25 00:00:58,440 --> 00:01:03,239 and human-made disasters sudden health 26 00:01:00,660 --> 00:01:06,199 problems or injuries and responding to 27 00:01:03,239 --> 00:01:06,199 thoughts of suicide 28 00:01:07,920 --> 00:01:14,220 so what would you do if you 29 00:01:11,119 --> 00:01:15,900 hypothetically woke up checked slack 30 00:01:14,220 --> 00:01:17,400 while you were still in bed I know you 31 00:01:15,900 --> 00:01:19,860 wouldn't do this 32 00:01:17,400 --> 00:01:21,420 and you saw 33 00:01:19,860 --> 00:01:24,720 this 34 00:01:21,420 --> 00:01:27,180 this is a slack message from our AWS Tam 35 00:01:24,720 --> 00:01:30,000 which says can you please look into this 36 00:01:27,180 --> 00:01:33,299 case number blah your AWS account number 37 00:01:30,000 --> 00:01:35,700 blah is compromised 38 00:01:33,299 --> 00:01:37,380 and since you don't have your AWS 39 00:01:35,700 --> 00:01:39,060 account ID is memorized and if you do 40 00:01:37,380 --> 00:01:41,520 please don't tell me 41 00:01:39,060 --> 00:01:44,180 you're mostly a sleep brain assumes it's 42 00:01:41,520 --> 00:01:44,180 the big account 43 00:01:45,659 --> 00:01:50,399 this is what my brain did 44 00:01:47,700 --> 00:01:53,540 gonna need some coffee 45 00:01:50,399 --> 00:01:53,540 it's like seven in the morning 46 00:01:53,700 --> 00:01:56,939 but seriously if this happened at your 47 00:01:55,380 --> 00:01:59,280 organization do you know what you would 48 00:01:56,939 --> 00:02:01,439 do is there someone you could call do 49 00:01:59,280 --> 00:02:03,060 you know how you would reach them do you 50 00:02:01,439 --> 00:02:05,840 know the answer to that now or would you 51 00:02:03,060 --> 00:02:05,840 need to look it up 52 00:02:06,600 --> 00:02:10,679 welcome to how to have an operational 53 00:02:08,220 --> 00:02:12,360 incident a crash course I'm Courtney I 54 00:02:10,679 --> 00:02:13,860 use sheer they pronouns and I'm giving 55 00:02:12,360 --> 00:02:15,599 this presentation from the unseated 56 00:02:13,860 --> 00:02:17,760 traditional territories of the musqueam 57 00:02:15,599 --> 00:02:19,200 Squamish and slay with youth nations in 58 00:02:17,760 --> 00:02:21,000 the city named Vancouver British 59 00:02:19,200 --> 00:02:22,800 Columbia by settlers 60 00:02:21,000 --> 00:02:24,180 I'm a career in management coach and an 61 00:02:22,800 --> 00:02:25,500 incident response specialist and I'm 62 00:02:24,180 --> 00:02:28,819 here to talk to you about what to do 63 00:02:25,500 --> 00:02:28,819 when everything goes wrong 64 00:02:29,940 --> 00:02:33,900 fundamentally what we're talking about 65 00:02:31,680 --> 00:02:35,879 is an emergency the most common 66 00:02:33,900 --> 00:02:38,640 definition of an emergency is a 67 00:02:35,879 --> 00:02:41,400 situation that posts poses an immediate 68 00:02:38,640 --> 00:02:43,739 risk to health life property or 69 00:02:41,400 --> 00:02:45,420 environment depending on your field any 70 00:02:43,739 --> 00:02:47,340 of those could apply to you but property 71 00:02:45,420 --> 00:02:49,700 or business reputation is probably the 72 00:02:47,340 --> 00:02:49,700 most common 73 00:02:51,060 --> 00:02:54,780 a lot of times I hear co-workers and 74 00:02:53,220 --> 00:02:57,599 like Folks at other conferences and 75 00:02:54,780 --> 00:02:59,160 companies say nobody's going to die it's 76 00:02:57,599 --> 00:03:00,599 not that bad we can calm down a little 77 00:02:59,160 --> 00:03:03,480 bit 78 00:03:00,599 --> 00:03:04,920 but do you know that let's say for the 79 00:03:03,480 --> 00:03:06,959 sake of argument that you actually do 80 00:03:04,920 --> 00:03:08,879 know who all of your customers are you 81 00:03:06,959 --> 00:03:11,099 don't but let's pretend for a moment do 82 00:03:08,879 --> 00:03:12,900 you know who their customers are and do 83 00:03:11,099 --> 00:03:15,420 you know all the workloads all of those 84 00:03:12,900 --> 00:03:17,400 people are doing do you know that all of 85 00:03:15,420 --> 00:03:19,379 them are only using your system in the 86 00:03:17,400 --> 00:03:20,940 expected ways 87 00:03:19,379 --> 00:03:22,680 there's no way to know the fourth or 88 00:03:20,940 --> 00:03:24,420 fifth order impacts of an outage of your 89 00:03:22,680 --> 00:03:25,860 system so saying that no one will die as 90 00:03:24,420 --> 00:03:28,080 a result of your outages making an 91 00:03:25,860 --> 00:03:29,459 assumption which might be unwarranted I 92 00:03:28,080 --> 00:03:31,379 don't say this because I want you to 93 00:03:29,459 --> 00:03:32,640 freeze but because I want you and 94 00:03:31,379 --> 00:03:34,500 everyone you work with to think 95 00:03:32,640 --> 00:03:36,060 seriously about incidents it doesn't 96 00:03:34,500 --> 00:03:37,620 make things better to have a loose and 97 00:03:36,060 --> 00:03:39,840 poorly specified incident response 98 00:03:37,620 --> 00:03:41,580 protocol even for very small incidents 99 00:03:39,840 --> 00:03:44,540 that are easily corrected or that only a 100 00:03:41,580 --> 00:03:44,540 few customers notice 101 00:03:46,140 --> 00:03:48,959 possibly you've heard productivity 102 00:03:47,700 --> 00:03:50,340 people talk about urgent versus 103 00:03:48,959 --> 00:03:52,500 important 104 00:03:50,340 --> 00:03:54,599 urgent is about timeline if something is 105 00:03:52,500 --> 00:03:55,799 urgent it needs to be handled quickly do 106 00:03:54,599 --> 00:03:56,879 you want to order lunch with your office 107 00:03:55,799 --> 00:03:59,220 friends 108 00:03:56,879 --> 00:04:00,659 important is about needs or Consequences 109 00:03:59,220 --> 00:04:02,220 making sure that you have your 110 00:04:00,659 --> 00:04:03,360 medications refilled or that you know 111 00:04:02,220 --> 00:04:07,340 where your children or your pets are 112 00:04:03,360 --> 00:04:07,340 important can be about Danger 113 00:04:08,400 --> 00:04:12,239 so an emergency is both urgent time 114 00:04:10,560 --> 00:04:14,700 sensitive and important it's about 115 00:04:12,239 --> 00:04:16,620 danger or Consequences a fire would be 116 00:04:14,700 --> 00:04:18,600 about about both day during consequences 117 00:04:16,620 --> 00:04:20,400 a broken bone from a bad fall would be 118 00:04:18,600 --> 00:04:22,740 about consequences 119 00:04:20,400 --> 00:04:24,919 but how do we know that we have an 120 00:04:22,740 --> 00:04:24,919 emergency 121 00:04:26,040 --> 00:04:30,900 someone a human has to know about it 122 00:04:28,919 --> 00:04:33,419 assess the urgency and the importance 123 00:04:30,900 --> 00:04:34,680 and decide that it is if a tree falls in 124 00:04:33,419 --> 00:04:36,300 the forest and there's no one there to 125 00:04:34,680 --> 00:04:38,880 witness it it's not an emergency because 126 00:04:36,300 --> 00:04:40,560 no one decided it was an emergency 127 00:04:38,880 --> 00:04:43,139 that means that someone needs to find 128 00:04:40,560 --> 00:04:45,120 out about it in the physical War World 129 00:04:43,139 --> 00:04:47,639 maybe you see something happen or come 130 00:04:45,120 --> 00:04:49,139 across someone in distress in the 131 00:04:47,639 --> 00:04:51,360 software industry that probably means 132 00:04:49,139 --> 00:04:53,580 you get paged hopefully by a monitoring 133 00:04:51,360 --> 00:04:55,320 surface or maybe by your support staff 134 00:04:53,580 --> 00:04:57,419 and then you have to decide how to 135 00:04:55,320 --> 00:04:59,580 respond in the physical world you would 136 00:04:57,419 --> 00:05:03,000 call Triple zero or maybe an alarm 137 00:04:59,580 --> 00:05:04,860 company or a security guard in the 138 00:05:03,000 --> 00:05:07,100 software industry maybe you page more 139 00:05:04,860 --> 00:05:07,100 people 140 00:05:08,400 --> 00:05:13,860 how many times have you said I didn't 141 00:05:12,060 --> 00:05:16,380 have time to think 142 00:05:13,860 --> 00:05:17,520 or how many times has someone said it to 143 00:05:16,380 --> 00:05:19,800 you 144 00:05:17,520 --> 00:05:21,540 thinking takes time 145 00:05:19,800 --> 00:05:23,880 and if you don't know what the options 146 00:05:21,540 --> 00:05:25,020 are while you're trying to think you'll 147 00:05:23,880 --> 00:05:27,360 freeze 148 00:05:25,020 --> 00:05:29,340 so we need to eliminate the need to 149 00:05:27,360 --> 00:05:31,440 think for as many people involved in the 150 00:05:29,340 --> 00:05:33,180 emergency as possible that's why when 151 00:05:31,440 --> 00:05:34,560 you call Triple zero it connects you to 152 00:05:33,180 --> 00:05:36,840 a dispatcher who can reach all the 153 00:05:34,560 --> 00:05:38,460 common response teams directly so you 154 00:05:36,840 --> 00:05:40,979 don't have to take a moment to decide 155 00:05:38,460 --> 00:05:43,400 who to call and then figure out how to 156 00:05:40,979 --> 00:05:43,400 reach them 157 00:05:45,960 --> 00:05:49,560 this is a picture of the sign on the 158 00:05:47,820 --> 00:05:50,820 back of my hotel room door in Toronto 159 00:05:49,560 --> 00:05:53,400 which is where I first gave this talk 160 00:05:50,820 --> 00:05:55,320 hotels across North America and probably 161 00:05:53,400 --> 00:05:57,360 in other places too have signs like this 162 00:05:55,320 --> 00:05:58,979 in every room just in case you should 163 00:05:57,360 --> 00:05:59,820 need to call for help or evacuate the 164 00:05:58,979 --> 00:06:02,340 building 165 00:05:59,820 --> 00:06:03,780 both the US and Canada use 9-1-1 as an 166 00:06:02,340 --> 00:06:06,479 emergency number where Australia has 167 00:06:03,780 --> 00:06:08,580 multiple numbers the triple zero one one 168 00:06:06,479 --> 00:06:10,139 two on mobile phones and 106 if you have 169 00:06:08,580 --> 00:06:12,000 a teletype machine 170 00:06:10,139 --> 00:06:14,100 emergency phone numbers create a 171 00:06:12,000 --> 00:06:15,660 framework for a response they represent 172 00:06:14,100 --> 00:06:19,320 years of planning and communication 173 00:06:15,660 --> 00:06:21,360 which includes these signs 174 00:06:19,320 --> 00:06:23,880 when you call Triple zero the dispatcher 175 00:06:21,360 --> 00:06:26,160 knows who they can call how to reach 176 00:06:23,880 --> 00:06:28,039 them how to get information from you 177 00:06:26,160 --> 00:06:30,840 about the situation you're reporting on 178 00:06:28,039 --> 00:06:32,699 and to remind you to keep yourself safe 179 00:06:30,840 --> 00:06:34,680 which is critical 180 00:06:32,699 --> 00:06:37,080 this framework enables you to respond 181 00:06:34,680 --> 00:06:38,460 when you encounter an emergency if and 182 00:06:37,080 --> 00:06:41,220 if you don't know what the options are 183 00:06:38,460 --> 00:06:43,020 for help you don't know what to do so 184 00:06:41,220 --> 00:06:45,120 this like primes you right it gets you 185 00:06:43,020 --> 00:06:46,740 ready to respond when you see something 186 00:06:45,120 --> 00:06:49,160 that you is unexpected and may be 187 00:06:46,740 --> 00:06:49,160 dangerous 188 00:06:49,979 --> 00:06:53,460 calling triple zero also triggers the 189 00:06:52,199 --> 00:06:55,020 response framework used by the 190 00:06:53,460 --> 00:06:56,699 dispatcher and emergency response 191 00:06:55,020 --> 00:06:58,919 specialist That's fire Personnel 192 00:06:56,699 --> 00:07:01,380 emergency medical technicians and police 193 00:06:58,919 --> 00:07:03,539 they all have plans for how to reach the 194 00:07:01,380 --> 00:07:05,220 emergency which is called assembling how 195 00:07:03,539 --> 00:07:06,720 to stay in contact with the dispatcher 196 00:07:05,220 --> 00:07:08,940 while they do that 197 00:07:06,720 --> 00:07:09,900 how to assess the emergency when they 198 00:07:08,940 --> 00:07:11,580 arrive 199 00:07:09,900 --> 00:07:13,680 how to communicate within their group 200 00:07:11,580 --> 00:07:14,880 and across groups at a firefighter 201 00:07:13,680 --> 00:07:16,199 Personnel have to coordinate with 202 00:07:14,880 --> 00:07:18,060 medical personnel so they need 203 00:07:16,199 --> 00:07:20,520 Intergroup comms 204 00:07:18,060 --> 00:07:22,919 how to escalate summon more fire trucks 205 00:07:20,520 --> 00:07:24,479 some in more medical personnel the 206 00:07:22,919 --> 00:07:26,520 details of these Frameworks vary by 207 00:07:24,479 --> 00:07:28,680 country Australia has published a PDF 208 00:07:26,520 --> 00:07:31,080 document called the crisis appreciation 209 00:07:28,680 --> 00:07:32,639 and strategic planning guidebook that 210 00:07:31,080 --> 00:07:33,780 discusses the high-level approach and I 211 00:07:32,639 --> 00:07:35,160 have a link to that at the end of my 212 00:07:33,780 --> 00:07:38,120 slides which the slides will be 213 00:07:35,160 --> 00:07:38,120 available after this talk 214 00:07:39,120 --> 00:07:43,380 so far what have we learned 215 00:07:41,759 --> 00:07:44,520 there's a lot of text on the slide don't 216 00:07:43,380 --> 00:07:46,199 worry about reading at all we're going 217 00:07:44,520 --> 00:07:48,419 to revisit each part 218 00:07:46,199 --> 00:07:50,520 Frameworks are about organization and 219 00:07:48,419 --> 00:07:52,259 planning if you discover something that 220 00:07:50,520 --> 00:07:54,000 might be an emergency and your mind goes 221 00:07:52,259 --> 00:07:56,520 blank that means that you haven't been 222 00:07:54,000 --> 00:07:58,500 enabled to respond you don't know what 223 00:07:56,520 --> 00:08:01,259 you need to know in order to figure out 224 00:07:58,500 --> 00:08:03,780 how to react 225 00:08:01,259 --> 00:08:07,020 an example of this that comes up fairly 226 00:08:03,780 --> 00:08:08,880 commonly especially recently if a friend 227 00:08:07,020 --> 00:08:10,919 comes to you and confesses that they 228 00:08:08,880 --> 00:08:13,380 have been having suicidal thoughts do 229 00:08:10,919 --> 00:08:15,780 you know what to do about that I find 230 00:08:13,380 --> 00:08:18,720 that a lot of the people that I talk to 231 00:08:15,780 --> 00:08:20,580 have to learn those skills specifically 232 00:08:18,720 --> 00:08:21,960 and there are programs and ways to learn 233 00:08:20,580 --> 00:08:23,879 those skills but the first few times it 234 00:08:21,960 --> 00:08:24,780 happens probably you don't know what to 235 00:08:23,879 --> 00:08:26,940 do 236 00:08:24,780 --> 00:08:28,800 responding to a security issue in the 237 00:08:26,940 --> 00:08:30,180 physical world like at your apartment 238 00:08:28,800 --> 00:08:33,479 building if you live in an apartment 239 00:08:30,180 --> 00:08:34,979 building or at your workplace those are 240 00:08:33,479 --> 00:08:36,120 also things where sometimes people just 241 00:08:34,979 --> 00:08:38,360 don't know what to do and they have to 242 00:08:36,120 --> 00:08:38,360 learn 243 00:08:39,839 --> 00:08:45,779 so let's talk about applying it 244 00:08:43,680 --> 00:08:47,279 now that we've sort of talked about what 245 00:08:45,779 --> 00:08:49,260 needs to happen and why it's important 246 00:08:47,279 --> 00:08:50,640 we can talk about what creating an 247 00:08:49,260 --> 00:08:52,500 incident response system for your 248 00:08:50,640 --> 00:08:54,180 organization would mean 249 00:08:52,500 --> 00:08:55,680 the best way to do this is to take a 250 00:08:54,180 --> 00:08:57,540 national system and adapt it to your 251 00:08:55,680 --> 00:08:59,279 needs and do your tools the rest of this 252 00:08:57,540 --> 00:09:01,580 talk is going to be guidance for doing 253 00:08:59,279 --> 00:09:01,580 that 254 00:09:02,580 --> 00:09:06,959 you need to know you need a way to know 255 00:09:04,920 --> 00:09:08,640 that there's something wrong so you need 256 00:09:06,959 --> 00:09:10,380 something that will alert you when 257 00:09:08,640 --> 00:09:12,000 there's an emergency in software this 258 00:09:10,380 --> 00:09:13,680 usually means that you need monitoring 259 00:09:12,000 --> 00:09:16,019 and alerting that you have some 260 00:09:13,680 --> 00:09:18,779 confidence in some companies do still 261 00:09:16,019 --> 00:09:20,220 use a network operations control center 262 00:09:18,779 --> 00:09:21,899 sort of arrangement where there are 263 00:09:20,220 --> 00:09:25,140 people watching graphs as opposed to 264 00:09:21,899 --> 00:09:27,180 automatic monitoring that can be useful 265 00:09:25,140 --> 00:09:28,920 but is generally slower than having a 266 00:09:27,180 --> 00:09:30,839 computer trigger something based on a 267 00:09:28,920 --> 00:09:32,399 threshold it might or might not reduce 268 00:09:30,839 --> 00:09:33,500 false positives it's a little hard to 269 00:09:32,399 --> 00:09:37,019 say 270 00:09:33,500 --> 00:09:39,240 monitoring is like the sort of basic 271 00:09:37,019 --> 00:09:41,580 best practice in North America but it 272 00:09:39,240 --> 00:09:44,700 differs over over the world how people 273 00:09:41,580 --> 00:09:46,320 what people consider to be the Baseline 274 00:09:44,700 --> 00:09:48,720 then you need to decide what constitutes 275 00:09:46,320 --> 00:09:49,980 an emergency for your organization in a 276 00:09:48,720 --> 00:09:51,480 commercial context this will usually 277 00:09:49,980 --> 00:09:52,740 mean that people either can't use what 278 00:09:51,480 --> 00:09:54,540 they're paying you for 279 00:09:52,740 --> 00:09:57,360 or they can't pay you in order to start 280 00:09:54,540 --> 00:09:59,820 using it one example would be if you 281 00:09:57,360 --> 00:10:02,880 can't buy things on Amazon Amazon gets 282 00:09:59,820 --> 00:10:04,800 really upset about that but also if you 283 00:10:02,880 --> 00:10:06,899 can't create a new social media post on 284 00:10:04,800 --> 00:10:09,300 whichever social media Network we're 285 00:10:06,899 --> 00:10:12,260 using today that's usually a big problem 286 00:10:09,300 --> 00:10:12,260 for that company 287 00:10:15,240 --> 00:10:19,019 like triple zero if you make people 288 00:10:17,100 --> 00:10:20,700 decide who to engage or look up how to 289 00:10:19,019 --> 00:10:23,100 engage them you'll waste precious time 290 00:10:20,700 --> 00:10:24,360 so you need to make sure that you have a 291 00:10:23,100 --> 00:10:26,459 single point of contact within your 292 00:10:24,360 --> 00:10:28,140 organization that it's easy to use and 293 00:10:26,459 --> 00:10:30,300 remember and you need to publicize it 294 00:10:28,140 --> 00:10:33,120 widely some companies use chat up 295 00:10:30,300 --> 00:10:34,980 commands in slack that's the most common 296 00:10:33,120 --> 00:10:36,899 and I think the quickest thing that I've 297 00:10:34,980 --> 00:10:39,240 seen sometimes it's a conference call 298 00:10:36,899 --> 00:10:41,220 number hopefully it's like a short code 299 00:10:39,240 --> 00:10:43,920 instead of having to type out a log 300 00:10:41,220 --> 00:10:45,720 number and remember a long number 301 00:10:43,920 --> 00:10:47,279 the people that you reach in this way 302 00:10:45,720 --> 00:10:48,899 should be able to fill the role of the 303 00:10:47,279 --> 00:10:51,120 dispatcher figuring out what the issue 304 00:10:48,899 --> 00:10:53,040 is who else to engage and how to reach 305 00:10:51,120 --> 00:10:54,779 them you probably want some technology 306 00:10:53,040 --> 00:10:56,880 to help with this this is a pain point 307 00:10:54,779 --> 00:11:00,839 in a lot of companies where it's hard to 308 00:10:56,880 --> 00:11:02,579 know okay so if the problem is the Kafka 309 00:11:00,839 --> 00:11:05,220 cluster who owns the Kafka cluster and 310 00:11:02,579 --> 00:11:06,660 how do we reach that team and how do we 311 00:11:05,220 --> 00:11:09,000 figure out which person on that team we 312 00:11:06,660 --> 00:11:11,760 should reach uh it's good to have some 313 00:11:09,000 --> 00:11:14,339 kind of directory of 314 00:11:11,760 --> 00:11:16,620 um on-call shortcuts and ways to page 315 00:11:14,339 --> 00:11:18,600 like on call person directly instead of 316 00:11:16,620 --> 00:11:19,680 having to work all of that out in the 317 00:11:18,600 --> 00:11:24,000 moment 318 00:11:19,680 --> 00:11:26,519 uh the people that you reach when you 319 00:11:24,000 --> 00:11:27,779 start this process are probably going to 320 00:11:26,519 --> 00:11:29,940 be the same people trained as incident 321 00:11:27,779 --> 00:11:32,640 commanders unlike the triple zero 322 00:11:29,940 --> 00:11:34,019 dispatcher uh triple zero dispatchers as 323 00:11:32,640 --> 00:11:36,300 far as I understand in Australia and 324 00:11:34,019 --> 00:11:37,680 definitely in North America they start 325 00:11:36,300 --> 00:11:40,320 their response but they don't continue 326 00:11:37,680 --> 00:11:42,300 managing it uh other people who go to 327 00:11:40,320 --> 00:11:43,680 the scene continue to manage it and most 328 00:11:42,300 --> 00:11:45,959 software companies it doesn't make sense 329 00:11:43,680 --> 00:11:47,700 to disconnect those two things because 330 00:11:45,959 --> 00:11:49,200 there aren't quite enough incidents to 331 00:11:47,700 --> 00:11:50,700 raise it to the point where you need a 332 00:11:49,200 --> 00:11:52,380 separate dispatcher from the person 333 00:11:50,700 --> 00:11:54,779 who's going to be managing the incident 334 00:11:52,380 --> 00:11:57,240 as it goes forward 335 00:11:54,779 --> 00:11:58,800 once you have an emergency this is 336 00:11:57,240 --> 00:12:00,839 really important 337 00:11:58,800 --> 00:12:02,579 once you have an emergency it can't be 338 00:12:00,839 --> 00:12:04,560 solved by one person emergency 339 00:12:02,579 --> 00:12:06,300 responders in the physical world are 340 00:12:04,560 --> 00:12:07,980 almost always sent out in pairs or in 341 00:12:06,300 --> 00:12:09,360 larger groups because when you have 342 00:12:07,980 --> 00:12:11,399 something that is both urgent and 343 00:12:09,360 --> 00:12:13,200 important you need the resources of 344 00:12:11,399 --> 00:12:15,540 multiple people and their skill sets to 345 00:12:13,200 --> 00:12:17,339 turn it around incident response is a 346 00:12:15,540 --> 00:12:19,019 team sport so that's why the first thing 347 00:12:17,339 --> 00:12:21,740 to do is to figure out how to get more 348 00:12:19,019 --> 00:12:21,740 people involved 349 00:12:23,160 --> 00:12:28,500 so here's another analysis or analogy 350 00:12:26,399 --> 00:12:30,540 between physical world and software 351 00:12:28,500 --> 00:12:32,279 spaces in the physical world where to go 352 00:12:30,540 --> 00:12:33,899 is sometimes ambiguous for instance 353 00:12:32,279 --> 00:12:35,360 national disasters won't necessarily 354 00:12:33,899 --> 00:12:37,800 have a street address that you can go to 355 00:12:35,360 --> 00:12:39,720 but usually responders will congregate 356 00:12:37,800 --> 00:12:41,220 in person and use radios or occasionally 357 00:12:39,720 --> 00:12:44,700 phones but relying on the phone network 358 00:12:41,220 --> 00:12:47,100 is not usually advised to coordinate 359 00:12:44,700 --> 00:12:49,079 among their group when they arrive 360 00:12:47,100 --> 00:12:50,760 you will probably need to decide on a 361 00:12:49,079 --> 00:12:52,200 digital site for your responders to join 362 00:12:50,760 --> 00:12:54,060 and that site will probably be the same 363 00:12:52,200 --> 00:12:55,380 as your Communications infrastructure it 364 00:12:54,060 --> 00:12:57,779 might be a conference call it might be 365 00:12:55,380 --> 00:12:59,279 slack the Specialists who join your 366 00:12:57,779 --> 00:13:01,380 incident response will also need to 367 00:12:59,279 --> 00:13:03,000 assemble at this site and they should 368 00:13:01,380 --> 00:13:04,860 not leave it without designating your 369 00:13:03,000 --> 00:13:06,839 replacement or without without being 370 00:13:04,860 --> 00:13:08,940 released by The Incident Commander 371 00:13:06,839 --> 00:13:10,620 make sure that this digital site 372 00:13:08,940 --> 00:13:12,180 whatever you're using is likely to still 373 00:13:10,620 --> 00:13:13,920 work if your own infrastructure is down 374 00:13:12,180 --> 00:13:16,200 and you need a contingency plan to think 375 00:13:13,920 --> 00:13:17,579 about okay so what if we're having a big 376 00:13:16,200 --> 00:13:19,860 incident and it turns out that it's 377 00:13:17,579 --> 00:13:21,540 because a lot of people are having a big 378 00:13:19,860 --> 00:13:24,000 incident and that means slack is down we 379 00:13:21,540 --> 00:13:26,720 need to have a fallback conference call 380 00:13:24,000 --> 00:13:26,720 number to use 381 00:13:28,620 --> 00:13:32,639 in software to assess the situation 382 00:13:30,959 --> 00:13:34,339 you'll probably be looking for impacted 383 00:13:32,639 --> 00:13:36,540 systems and customer experience 384 00:13:34,339 --> 00:13:37,680 identifying the impacted systems means 385 00:13:36,540 --> 00:13:39,360 that you can get the right people to 386 00:13:37,680 --> 00:13:41,220 join your incident response things like 387 00:13:39,360 --> 00:13:43,139 knowing whether it's the copper cluster 388 00:13:41,220 --> 00:13:44,940 or your backend database or whatever it 389 00:13:43,139 --> 00:13:47,120 is that you're using versus your 390 00:13:44,940 --> 00:13:49,019 front-end web server Fleet 391 00:13:47,120 --> 00:13:50,880 identifying the customer's experience 392 00:13:49,019 --> 00:13:52,440 means you can tell customers what issues 393 00:13:50,880 --> 00:13:53,940 you are aware of so that they know that 394 00:13:52,440 --> 00:13:55,500 you're working on it and so they will 395 00:13:53,940 --> 00:13:56,820 still open support cases for other 396 00:13:55,500 --> 00:13:58,800 issues 397 00:13:56,820 --> 00:14:00,540 this is an ongoing task for The Incident 398 00:13:58,800 --> 00:14:02,040 Commander depending on the issue the 399 00:14:00,540 --> 00:14:04,139 answers to these questions could change 400 00:14:02,040 --> 00:14:07,079 during the course of your response based 401 00:14:04,139 --> 00:14:08,519 on the environment or your actions you 402 00:14:07,079 --> 00:14:10,980 could bring up a part of the system that 403 00:14:08,519 --> 00:14:12,660 was not working before and change what 404 00:14:10,980 --> 00:14:16,100 is available to customers and what you 405 00:14:12,660 --> 00:14:16,100 and your team need to be working on 406 00:14:16,980 --> 00:14:21,360 the essence of successful incident 407 00:14:19,500 --> 00:14:23,639 response is cooperation and delegation 408 00:14:21,360 --> 00:14:25,019 no one can handle this solo as 409 00:14:23,639 --> 00:14:26,519 previously covered so that means 410 00:14:25,019 --> 00:14:28,680 dividing up the work 411 00:14:26,519 --> 00:14:30,180 you need to consider that you want to 412 00:14:28,680 --> 00:14:31,800 avoid duplication of effort when you're 413 00:14:30,180 --> 00:14:34,339 dividing up the work and you want to 414 00:14:31,800 --> 00:14:36,540 paralyze parallelize work on the problem 415 00:14:34,339 --> 00:14:38,459 sometimes you are running multiple 416 00:14:36,540 --> 00:14:40,500 investigations side by side because you 417 00:14:38,459 --> 00:14:42,779 don't know what the problem is yet and 418 00:14:40,500 --> 00:14:44,519 it wastes less time than doing them one 419 00:14:42,779 --> 00:14:46,199 after the other 420 00:14:44,519 --> 00:14:47,880 this also means that all the people 421 00:14:46,199 --> 00:14:49,500 handling this work need to report back 422 00:14:47,880 --> 00:14:52,760 to The Incident Commander on a regular 423 00:14:49,500 --> 00:14:52,760 basis to keep them updated 424 00:14:53,639 --> 00:14:57,240 for the duration of the incident The 425 00:14:55,980 --> 00:14:58,860 Incident Commander should be the start 426 00:14:57,240 --> 00:15:00,600 of Authority for the company 427 00:14:58,860 --> 00:15:02,220 The Incident Commander is the only one 428 00:15:00,600 --> 00:15:03,720 with a view of the whole problem and all 429 00:15:02,220 --> 00:15:05,699 the people working on it because their 430 00:15:03,720 --> 00:15:08,160 focus is the progress of the 431 00:15:05,699 --> 00:15:09,300 investigation and the attempt to fix the 432 00:15:08,160 --> 00:15:11,160 problem 433 00:15:09,300 --> 00:15:12,480 even if everyone is working on things in 434 00:15:11,160 --> 00:15:14,459 the same slack channel the incident 435 00:15:12,480 --> 00:15:16,620 Commander's only job is to pay attention 436 00:15:14,459 --> 00:15:18,899 to what do we understand the problem to 437 00:15:16,620 --> 00:15:20,519 be right now what are we trying in order 438 00:15:18,899 --> 00:15:23,100 to make sure that our understanding is 439 00:15:20,519 --> 00:15:24,600 accurate what are we going to do if our 440 00:15:23,100 --> 00:15:26,399 understanding is accurate in order to 441 00:15:24,600 --> 00:15:28,199 get things fixed 442 00:15:26,399 --> 00:15:29,579 this means that it's critical that no 443 00:15:28,199 --> 00:15:31,500 one argue with or counter man The 444 00:15:29,579 --> 00:15:33,959 Incident Commander not other responders 445 00:15:31,500 --> 00:15:36,000 and not managers or executives 446 00:15:33,959 --> 00:15:37,560 if someone argues it means that everyone 447 00:15:36,000 --> 00:15:39,959 helping with the incident has to decide 448 00:15:37,560 --> 00:15:41,100 whose side to take in that moment are 449 00:15:39,959 --> 00:15:42,720 they going to follow what The Incident 450 00:15:41,100 --> 00:15:45,180 Commander says or the person arguing 451 00:15:42,720 --> 00:15:48,420 with them this wastes time and it's 452 00:15:45,180 --> 00:15:50,699 counterproductive and it this this start 453 00:15:48,420 --> 00:15:52,320 of authority thing also means that no 454 00:15:50,699 --> 00:15:54,060 one leaves the incident until they are 455 00:15:52,320 --> 00:15:56,839 released by The Incident Commander or 456 00:15:54,060 --> 00:15:56,839 they're replaced 457 00:15:58,680 --> 00:16:02,940 you also need some criteria for deciding 458 00:16:00,899 --> 00:16:04,860 when things are done sometimes that's 459 00:16:02,940 --> 00:16:07,440 clear there's no more impact or we 460 00:16:04,860 --> 00:16:09,600 rolled back a bad release but it could 461 00:16:07,440 --> 00:16:12,060 be unclear sometimes what if the problem 462 00:16:09,600 --> 00:16:14,100 is we mitigated the pro whatever went 463 00:16:12,060 --> 00:16:16,560 wrong but we won't have a fix from the 464 00:16:14,100 --> 00:16:19,139 vendor for three weeks so it could go 465 00:16:16,560 --> 00:16:20,760 wrong again what if the issue is we 466 00:16:19,139 --> 00:16:22,260 can't mitigate the problem we won't have 467 00:16:20,760 --> 00:16:23,639 the fix from the vendor for three weeks 468 00:16:22,260 --> 00:16:25,380 we're just gonna be running in a 469 00:16:23,639 --> 00:16:27,720 degraded state for that time 470 00:16:25,380 --> 00:16:29,880 what if the issue is a security incident 471 00:16:27,720 --> 00:16:31,199 and now everyone in the company has to 472 00:16:29,880 --> 00:16:32,940 patch their reticence and it's going to 473 00:16:31,199 --> 00:16:34,740 take three weeks to do that 474 00:16:32,940 --> 00:16:36,839 start by coming up with a few high-level 475 00:16:34,740 --> 00:16:38,459 plans for cases like this when they 476 00:16:36,839 --> 00:16:39,480 happen you can use those plans as a 477 00:16:38,459 --> 00:16:41,459 starting point 478 00:16:39,480 --> 00:16:43,199 you can almost never come up with a 479 00:16:41,459 --> 00:16:44,759 truly exact plan for something that's 480 00:16:43,199 --> 00:16:46,920 going to happen in the future where you 481 00:16:44,759 --> 00:16:48,720 can just like run down that list this is 482 00:16:46,920 --> 00:16:51,120 where judgment and experience comes into 483 00:16:48,720 --> 00:16:52,680 play but if you have the plan you have 484 00:16:51,120 --> 00:16:54,360 some place to start and it keeps you 485 00:16:52,680 --> 00:16:56,639 from having like the blank page syndrome 486 00:16:54,360 --> 00:16:59,180 where you just stare at your brain isn't 487 00:16:56,639 --> 00:16:59,180 sure what to do 488 00:17:00,120 --> 00:17:04,380 finally there's a special case of 489 00:17:02,279 --> 00:17:06,720 assembly and dispersal which is shifts 490 00:17:04,380 --> 00:17:08,640 operational incidents can be very quick 491 00:17:06,720 --> 00:17:11,280 but they can also last hours or days 492 00:17:08,640 --> 00:17:14,040 security incidents sometimes last weeks 493 00:17:11,280 --> 00:17:15,480 you need an idea of what a shift is and 494 00:17:14,040 --> 00:17:17,819 what the process for a shift change 495 00:17:15,480 --> 00:17:20,280 should look like I recommend four hour 496 00:17:17,819 --> 00:17:22,140 shifts this is what we found that people 497 00:17:20,280 --> 00:17:24,000 could handle as Incident Commander 498 00:17:22,140 --> 00:17:26,040 empirically like people would just 499 00:17:24,000 --> 00:17:27,839 really get tired and their response time 500 00:17:26,040 --> 00:17:30,720 would really slow after about four hours 501 00:17:27,839 --> 00:17:32,820 and four hours is also what I learned in 502 00:17:30,720 --> 00:17:34,740 the classes given by professional fire 503 00:17:32,820 --> 00:17:37,100 Personnel on how they do Incident 504 00:17:34,740 --> 00:17:37,100 Management 505 00:17:38,280 --> 00:17:41,160 so if you like to take pictures of 506 00:17:39,900 --> 00:17:42,539 slides this is a good one to take 507 00:17:41,160 --> 00:17:43,679 pictures of but you don't have to 508 00:17:42,539 --> 00:17:45,840 because the slides are going to be 509 00:17:43,679 --> 00:17:48,179 available afterwards this is the summary 510 00:17:45,840 --> 00:17:49,440 this is a minimum viable incident 511 00:17:48,179 --> 00:17:51,179 response plan 512 00:17:49,440 --> 00:17:52,799 there's a lot more to it than this but 513 00:17:51,179 --> 00:17:54,900 this is how you can get started if you 514 00:17:52,799 --> 00:17:58,080 cover the stuff you're on your way and 515 00:17:54,900 --> 00:18:00,179 the stuff is is it an emergency 516 00:17:58,080 --> 00:18:02,340 do you have people who know the relevant 517 00:18:00,179 --> 00:18:03,720 subject areas do you have someone who 518 00:18:02,340 --> 00:18:05,340 knows how to organize them that would be 519 00:18:03,720 --> 00:18:07,080 the Incident Commander do you have a 520 00:18:05,340 --> 00:18:09,360 place to assemble do you have a way to 521 00:18:07,080 --> 00:18:11,039 communicate do you have an idea of what 522 00:18:09,360 --> 00:18:13,460 a shift is and do you have an idea of 523 00:18:11,039 --> 00:18:13,460 when it's over 524 00:18:15,179 --> 00:18:19,140 after that you need to train people 525 00:18:17,340 --> 00:18:20,760 people need to be able to practice this 526 00:18:19,140 --> 00:18:22,919 so that they don't freeze when the 527 00:18:20,760 --> 00:18:24,120 moment happens you need training and 528 00:18:22,919 --> 00:18:26,220 documentation for the incident 529 00:18:24,120 --> 00:18:27,720 commanders at the very Baris minimum 530 00:18:26,220 --> 00:18:28,740 even if you don't have time for anything 531 00:18:27,720 --> 00:18:31,200 else 532 00:18:28,740 --> 00:18:32,760 but you will get much better results if 533 00:18:31,200 --> 00:18:35,100 all of the people who might be patient 534 00:18:32,760 --> 00:18:37,380 to an incident are trained as if they 535 00:18:35,100 --> 00:18:39,840 were incident commanders themselves it 536 00:18:37,380 --> 00:18:41,220 will tell them they'll already know what 537 00:18:39,840 --> 00:18:44,039 the Incident Commander is going to ask 538 00:18:41,220 --> 00:18:47,880 what they why they're asking how it will 539 00:18:44,039 --> 00:18:50,100 help you will get responses often before 540 00:18:47,880 --> 00:18:52,380 they're requested and it reduces 541 00:18:50,100 --> 00:18:53,760 grumpiness if you feel like there's an 542 00:18:52,380 --> 00:18:55,820 emergency and someone's asking you for 543 00:18:53,760 --> 00:18:58,860 something that you think is completely 544 00:18:55,820 --> 00:19:00,720 unacceptable unreasonable beside the 545 00:18:58,860 --> 00:19:03,299 point then you get angry and you don't 546 00:19:00,720 --> 00:19:04,740 want to answer but if you understand the 547 00:19:03,299 --> 00:19:06,900 protocol that The Incident Commander is 548 00:19:04,740 --> 00:19:09,500 going through in their head it no longer 549 00:19:06,900 --> 00:19:12,660 feels inappropriate to have a question 550 00:19:09,500 --> 00:19:15,980 that is like surprising to you or the 551 00:19:12,660 --> 00:19:15,980 question might not even surprise you 552 00:19:16,860 --> 00:19:20,280 after that 553 00:19:18,539 --> 00:19:22,620 there's a lot of places that you can go 554 00:19:20,280 --> 00:19:24,780 these are what I just covered is the 555 00:19:22,620 --> 00:19:26,640 very beginning steps these are the next 556 00:19:24,780 --> 00:19:28,380 things that you could look at how do you 557 00:19:26,640 --> 00:19:30,059 communicate with customers well this is 558 00:19:28,380 --> 00:19:32,220 happening how do you engage your lawyers 559 00:19:30,059 --> 00:19:33,720 how do you engage your PR people how do 560 00:19:32,220 --> 00:19:35,460 you engage the executives to let them 561 00:19:33,720 --> 00:19:38,100 know that something is wrong how do you 562 00:19:35,460 --> 00:19:40,080 engage vendors if they are necessary to 563 00:19:38,100 --> 00:19:42,480 the solution and how do your vendors 564 00:19:40,080 --> 00:19:44,400 engage you if they're having incidents 565 00:19:42,480 --> 00:19:46,820 that might impact what's happening on 566 00:19:44,400 --> 00:19:46,820 your system 567 00:19:49,320 --> 00:19:53,760 so far I've spent all this time 568 00:19:52,200 --> 00:19:55,320 discussing the benefits of having or 569 00:19:53,760 --> 00:19:56,640 starting an incident response system but 570 00:19:55,320 --> 00:19:58,260 it's also important to talk about the 571 00:19:56,640 --> 00:20:00,179 drawbacks 572 00:19:58,260 --> 00:20:01,679 incident response especially in a 573 00:20:00,179 --> 00:20:02,820 corporate environment is a burnout 574 00:20:01,679 --> 00:20:04,620 Factory 575 00:20:02,820 --> 00:20:07,080 this is in large part because capitalism 576 00:20:04,620 --> 00:20:08,820 is a burnout Factory but it's worse than 577 00:20:07,080 --> 00:20:11,400 incident response fundamentally because 578 00:20:08,820 --> 00:20:13,080 incident response is a cost center it's 579 00:20:11,400 --> 00:20:14,880 not a profit Center 580 00:20:13,080 --> 00:20:17,100 that means that shareholder capitalism 581 00:20:14,880 --> 00:20:19,440 wants to have an incident response team 582 00:20:17,100 --> 00:20:21,179 when something bad has just happened but 583 00:20:19,440 --> 00:20:23,220 when something bad has not just happened 584 00:20:21,179 --> 00:20:25,679 it kind of wants to squeeze that team 585 00:20:23,220 --> 00:20:27,480 dry in service of better profits and run 586 00:20:25,679 --> 00:20:29,460 as lean as possible for as long as 587 00:20:27,480 --> 00:20:31,320 possible on the theory that probably 588 00:20:29,460 --> 00:20:33,539 things won't go wrong and if they do it 589 00:20:31,320 --> 00:20:35,940 probably won't be that bad 590 00:20:33,539 --> 00:20:37,620 if you work an incident response and you 591 00:20:35,940 --> 00:20:38,820 feel that you are being exploited for 592 00:20:37,620 --> 00:20:41,820 your willingness to do what the business 593 00:20:38,820 --> 00:20:44,100 needs and not what it actually wants 594 00:20:41,820 --> 00:20:45,900 you're right you are 595 00:20:44,100 --> 00:20:48,179 and the same goes for other people who 596 00:20:45,900 --> 00:20:50,280 work in low glamor underfunded keeping 597 00:20:48,179 --> 00:20:51,840 the lights on departments shareholder 598 00:20:50,280 --> 00:20:53,520 capitalism relies on people being 599 00:20:51,840 --> 00:20:54,960 willing to wear themselves out for the 600 00:20:53,520 --> 00:20:57,059 good of the company and the customers 601 00:20:54,960 --> 00:20:59,600 this is especially true in incident 602 00:20:57,059 --> 00:20:59,600 response 603 00:21:00,720 --> 00:21:04,500 is it possible to make real change in 604 00:21:03,120 --> 00:21:06,900 this situation 605 00:21:04,500 --> 00:21:09,660 sometimes it's it's sometimes possible 606 00:21:06,900 --> 00:21:12,059 to do a corporate activism and convince 607 00:21:09,660 --> 00:21:13,740 the managers for a while the incident 608 00:21:12,059 --> 00:21:16,140 response should be funded 609 00:21:13,740 --> 00:21:18,419 however in a publicly traded company as 610 00:21:16,140 --> 00:21:20,100 previously covered the people that you 611 00:21:18,419 --> 00:21:22,380 would truly have to convince to make 612 00:21:20,100 --> 00:21:25,020 this stick in the long term would be the 613 00:21:22,380 --> 00:21:27,299 shareholders and that's not possible 614 00:21:25,020 --> 00:21:29,340 structurally employees don't really have 615 00:21:27,299 --> 00:21:30,919 access to the shareholders and the 616 00:21:29,340 --> 00:21:33,539 shareholders aren't thinking about 617 00:21:30,919 --> 00:21:35,760 long-term value and long-term livability 618 00:21:33,539 --> 00:21:39,000 of people's jobs they're thinking about 619 00:21:35,760 --> 00:21:41,580 short to medium term profit 620 00:21:39,000 --> 00:21:44,039 if you've heard the saying never waste a 621 00:21:41,580 --> 00:21:46,020 disaster this advice is about using the 622 00:21:44,039 --> 00:21:48,179 fear caused by a major incident or a 623 00:21:46,020 --> 00:21:50,220 near-miss to try to convince Executives 624 00:21:48,179 --> 00:21:51,600 to fund incident response generally 625 00:21:50,220 --> 00:21:53,940 speaking 626 00:21:51,600 --> 00:21:56,520 but trying to create systemic change via 627 00:21:53,940 --> 00:21:57,960 immediate fear isn't very effective and 628 00:21:56,520 --> 00:22:00,600 while you might be able to temporarily 629 00:21:57,960 --> 00:22:01,799 convince an executive the shareholders 630 00:22:00,600 --> 00:22:03,539 are there to make sure that the 631 00:22:01,799 --> 00:22:05,520 executives keep shareholder value as 632 00:22:03,539 --> 00:22:07,799 their North Star and shareholder value 633 00:22:05,520 --> 00:22:10,820 is widely regarded as antithetical to 634 00:22:07,799 --> 00:22:10,820 funding a cost center 635 00:22:12,000 --> 00:22:17,700 what if someone hands you the Hot Potato 636 00:22:15,000 --> 00:22:19,440 sometimes after a big incident a person 637 00:22:17,700 --> 00:22:22,200 is tasked with standing up an incident 638 00:22:19,440 --> 00:22:24,780 response framework or fixing it up if 639 00:22:22,200 --> 00:22:26,220 that's you it's probably not great for 640 00:22:24,780 --> 00:22:27,960 you politically 641 00:22:26,220 --> 00:22:30,539 I would recommend assembling something 642 00:22:27,960 --> 00:22:32,580 that's like medium thorough covers all 643 00:22:30,539 --> 00:22:34,740 the obvious cases has a list of ways to 644 00:22:32,580 --> 00:22:36,480 expand it sort of like here's what we 645 00:22:34,740 --> 00:22:38,159 did and here's what We're Dreaming and 646 00:22:36,480 --> 00:22:39,600 here's the people that we would need to 647 00:22:38,159 --> 00:22:41,640 get that done 648 00:22:39,600 --> 00:22:43,740 and once you have that I'd recommend 649 00:22:41,640 --> 00:22:45,480 getting out of there 650 00:22:43,740 --> 00:22:47,940 this kind of thing is usually handed to 651 00:22:45,480 --> 00:22:50,700 either the organizational sacrificial 652 00:22:47,940 --> 00:22:52,919 goat or to The Prodigal guy 653 00:22:50,700 --> 00:22:55,559 and it helps to know which one you are 654 00:22:52,919 --> 00:22:57,960 if it's you if you're the prodigal guy 655 00:22:55,559 --> 00:22:59,340 sticking around too long means that 656 00:22:57,960 --> 00:23:01,500 you're likely to become the sacrificial 657 00:22:59,340 --> 00:23:03,120 goat so keep that in mind and I did say 658 00:23:01,500 --> 00:23:06,059 guy because 659 00:23:03,120 --> 00:23:08,280 under capitalism the person who is 660 00:23:06,059 --> 00:23:11,760 expected to perform great things and and 661 00:23:08,280 --> 00:23:14,159 bring new ideas to the organization is 662 00:23:11,760 --> 00:23:17,700 often someone who people think of as a 663 00:23:14,159 --> 00:23:19,860 guy and who Executives find remind them 664 00:23:17,700 --> 00:23:22,400 of themselves at that age whatever age 665 00:23:19,860 --> 00:23:22,400 that is 666 00:23:22,980 --> 00:23:28,980 and that's pretty depressing 667 00:23:26,460 --> 00:23:31,140 it's not a great place to be working in 668 00:23:28,980 --> 00:23:33,179 most companies most of the time 669 00:23:31,140 --> 00:23:36,179 the only way I really see to change this 670 00:23:33,179 --> 00:23:38,640 is regulation because the thing that can 671 00:23:36,179 --> 00:23:40,620 cause a company to do something that is 672 00:23:38,640 --> 00:23:44,960 against its short to medium term profit 673 00:23:40,620 --> 00:23:44,960 goals is a requirement from outside 674 00:23:46,559 --> 00:23:50,820 but to return to the original story 675 00:23:49,080 --> 00:23:53,460 what actually happened with that message 676 00:23:50,820 --> 00:23:54,780 from our AWS Tam for those of you who 677 00:23:53,460 --> 00:23:56,760 are still wondering 678 00:23:54,780 --> 00:23:58,620 I 679 00:23:56,760 --> 00:24:00,780 found out that the account in question 680 00:23:58,620 --> 00:24:03,240 was a brand new developer account that 681 00:24:00,780 --> 00:24:05,100 had its creds checked into GitHub by 682 00:24:03,240 --> 00:24:07,260 accident first thing account creation 683 00:24:05,100 --> 00:24:08,940 creds checked into GitHub like same five 684 00:24:07,260 --> 00:24:11,460 minutes 685 00:24:08,940 --> 00:24:13,080 but several other people also heard that 686 00:24:11,460 --> 00:24:14,520 we had an account compromise that came 687 00:24:13,080 --> 00:24:17,580 to me about it over the course of the 688 00:24:14,520 --> 00:24:21,360 day so that was kind of 689 00:24:17,580 --> 00:24:25,760 fun annoying there was a lot of sort of 690 00:24:21,360 --> 00:24:25,760 emotional mop up after that moment 691 00:24:27,179 --> 00:24:32,220 uh if you have some further resources 692 00:24:28,980 --> 00:24:33,840 here uh the Wikipedia page on the 693 00:24:32,220 --> 00:24:35,280 incident response system oh I'm sorry 694 00:24:33,840 --> 00:24:37,620 there's some unprintable characters in 695 00:24:35,280 --> 00:24:39,120 the slide that should not be there uh I 696 00:24:37,620 --> 00:24:42,659 will clean those up before I release the 697 00:24:39,120 --> 00:24:45,480 slides the Wikipedia page is pretty good 698 00:24:42,659 --> 00:24:46,980 at like helping you get oriented in like 699 00:24:45,480 --> 00:24:48,900 what are we trying to cover and why do 700 00:24:46,980 --> 00:24:50,100 we think it's important there's a book 701 00:24:48,900 --> 00:24:53,700 called Incident Management for 702 00:24:50,100 --> 00:24:55,500 operations by Rob schnepp Rod Vidal and 703 00:24:53,700 --> 00:24:58,320 Chris Hawley and those are the folks who 704 00:24:55,500 --> 00:25:00,260 trained me who are former firefighters 705 00:24:58,320 --> 00:25:02,880 that I mentioned earlier 706 00:25:00,260 --> 00:25:04,880 the guidebook from the Australian 707 00:25:02,880 --> 00:25:07,679 government I have the link to that here 708 00:25:04,880 --> 00:25:09,240 and also a Paradise built in Hell the 709 00:25:07,679 --> 00:25:11,039 extraordinary communities that arise in 710 00:25:09,240 --> 00:25:14,220 disaster is a book written by Rebecca 711 00:25:11,039 --> 00:25:15,900 solnit and she did a lot of sociological 712 00:25:14,220 --> 00:25:18,299 research about how regular people 713 00:25:15,900 --> 00:25:19,940 respond during earthquake during it 714 00:25:18,299 --> 00:25:23,760 after earthquakes and 715 00:25:19,940 --> 00:25:25,500 wildfires and other major disasters 716 00:25:23,760 --> 00:25:28,380 it was a fascinating book 717 00:25:25,500 --> 00:25:30,600 and actually a very hopeful book 718 00:25:28,380 --> 00:25:32,100 some things that I want to read the 719 00:25:30,600 --> 00:25:34,140 survivors Club the secrets in science 720 00:25:32,100 --> 00:25:36,840 that could save your life I've been told 721 00:25:34,140 --> 00:25:39,419 that it talks through like how people 722 00:25:36,840 --> 00:25:41,279 survive Aviation accidents and how 723 00:25:39,419 --> 00:25:42,720 people don't survive it and that's 724 00:25:41,279 --> 00:25:45,600 something that I'm interested in hearing 725 00:25:42,720 --> 00:25:47,880 about also there is a doorstop book 726 00:25:45,600 --> 00:25:50,460 called The Challenger launch decision 727 00:25:47,880 --> 00:25:52,679 risky technology culture and deviance at 728 00:25:50,460 --> 00:25:56,159 Nasa by Diane Vaughn she's a sociologist 729 00:25:52,679 --> 00:25:58,440 and she went through all of the piles of 730 00:25:56,159 --> 00:26:00,179 paper after the Challenger disaster and 731 00:25:58,440 --> 00:26:01,860 brought up this huge book with her 732 00:26:00,179 --> 00:26:03,059 findings that I've heard very good 733 00:26:01,860 --> 00:26:05,039 things about have not had a chance to 734 00:26:03,059 --> 00:26:07,020 read yet 735 00:26:05,039 --> 00:26:08,580 couple of acknowledgments my friend 736 00:26:07,020 --> 00:26:10,559 Marlena Compton for reading over a draft 737 00:26:08,580 --> 00:26:12,539 of this presentation and providing 738 00:26:10,559 --> 00:26:15,539 encouragement that this was in fact a 739 00:26:12,539 --> 00:26:17,400 useful topic to be presenting on also 740 00:26:15,539 --> 00:26:19,679 Mental Health First Aid Canada which 741 00:26:17,400 --> 00:26:21,900 when I took that class really gave me 742 00:26:19,679 --> 00:26:23,220 the understanding that when you freeze 743 00:26:21,900 --> 00:26:25,799 an emergency it's because you're not 744 00:26:23,220 --> 00:26:27,480 sure what to do and if you have the 745 00:26:25,799 --> 00:26:29,460 protocol in your head beforehand then 746 00:26:27,480 --> 00:26:31,380 you can figure it out more easily there 747 00:26:29,460 --> 00:26:32,880 is a mental health first aid Australia I 748 00:26:31,380 --> 00:26:34,679 believe Mental Health First Aid first 749 00:26:32,880 --> 00:26:37,740 came from Australia so it's a really 750 00:26:34,679 --> 00:26:38,940 cool training and I recommend it to 751 00:26:37,740 --> 00:26:41,299 anyone who thinks they might be 752 00:26:38,940 --> 00:26:41,299 interested 753 00:26:41,460 --> 00:26:47,830 and thank you so much you can find me on 754 00:26:44,220 --> 00:26:53,710 the Discord or on Mastodon 755 00:26:47,830 --> 00:26:53,710 [Applause] 756 00:26:54,600 --> 00:26:58,020 thank you very much Courtney we have a 757 00:26:56,580 --> 00:26:59,400 few minutes for questions if you do have 758 00:26:58,020 --> 00:27:00,960 a question could I get you to raise your 759 00:26:59,400 --> 00:27:03,000 hand we have a runner who will bring a 760 00:27:00,960 --> 00:27:04,740 microphone to you uh while we get the 761 00:27:03,000 --> 00:27:06,360 room walked out warmed up and make sure 762 00:27:04,740 --> 00:27:08,159 we get questions for you 763 00:27:06,360 --> 00:27:10,080 um you mentioned training and and sort 764 00:27:08,159 --> 00:27:11,220 of get making sure everybody knows 765 00:27:10,080 --> 00:27:13,320 what's what they're supposed to do so 766 00:27:11,220 --> 00:27:15,360 that they respond well is there any 767 00:27:13,320 --> 00:27:17,340 value in sort of de-escalating this so 768 00:27:15,360 --> 00:27:19,320 it's a practice that is done more often 769 00:27:17,340 --> 00:27:21,120 rather than just when something is 770 00:27:19,320 --> 00:27:23,880 either literally or figuratively on fire 771 00:27:21,120 --> 00:27:26,100 and and go through the process either as 772 00:27:23,880 --> 00:27:27,419 mocks or with much much lower scale 773 00:27:26,100 --> 00:27:29,340 incidents 774 00:27:27,419 --> 00:27:30,960 yes you can sort of treat it like a 775 00:27:29,340 --> 00:27:32,340 tabletop role-playing game uh which 776 00:27:30,960 --> 00:27:35,039 turns out to be a ton of fun people 777 00:27:32,340 --> 00:27:36,779 really enjoy that it's also very hard to 778 00:27:35,039 --> 00:27:40,200 get funded in terms of like time and 779 00:27:36,779 --> 00:27:42,779 human attention uh and planning but if 780 00:27:40,200 --> 00:27:45,179 you can get it it's incredibly useful 781 00:27:42,779 --> 00:27:46,799 it's also very useful before like major 782 00:27:45,179 --> 00:27:48,419 launches if you're launching a whole new 783 00:27:46,799 --> 00:27:51,900 product and you're going to need to 784 00:27:48,419 --> 00:27:53,520 figure out how do we engage people for 785 00:27:51,900 --> 00:27:55,140 this new product what does it depend on 786 00:27:53,520 --> 00:27:57,720 who are we going to be working with it's 787 00:27:55,140 --> 00:27:59,580 a great great tactic okay 788 00:27:57,720 --> 00:28:01,760 uh do we have questions we have one here 789 00:27:59,580 --> 00:28:01,760 yes 790 00:28:02,460 --> 00:28:08,460 um great talk thanks very much 791 00:28:05,159 --> 00:28:11,039 um one um one place where I uh get some 792 00:28:08,460 --> 00:28:13,200 inspiration uh in this sort of space and 793 00:28:11,039 --> 00:28:15,840 this is um you know working both as a 794 00:28:13,200 --> 00:28:17,700 developer and in operations you know and 795 00:28:15,840 --> 00:28:19,679 and every developer should work in 796 00:28:17,700 --> 00:28:21,179 operations at least for a few years is 797 00:28:19,679 --> 00:28:23,279 the airline industry I don't know 798 00:28:21,179 --> 00:28:26,039 whether you've heard of the qf 32 799 00:28:23,279 --> 00:28:28,200 um scenario they have a concept called a 800 00:28:26,039 --> 00:28:29,640 just culture and I was wondering whether 801 00:28:28,200 --> 00:28:32,880 you're familiar with that and whether it 802 00:28:29,640 --> 00:28:34,740 sort of found its way into I.T and these 803 00:28:32,880 --> 00:28:36,720 various other places 804 00:28:34,740 --> 00:28:38,460 yeah I actually have another talk that 805 00:28:36,720 --> 00:28:41,220 I've given at other conferences 806 00:28:38,460 --> 00:28:44,039 um which I can post on messed on uh 807 00:28:41,220 --> 00:28:45,360 where I talk about how to have a 808 00:28:44,039 --> 00:28:46,679 blameless switch perspective and like 809 00:28:45,360 --> 00:28:48,900 how to talk about problems without 810 00:28:46,679 --> 00:28:50,700 pointing fingers and finding better 811 00:28:48,900 --> 00:28:53,640 Solutions collaboratively so yeah just 812 00:28:50,700 --> 00:28:55,620 culture is really a great 813 00:28:53,640 --> 00:28:56,880 um framework 814 00:28:55,620 --> 00:28:58,260 okay I think we have another question at 815 00:28:56,880 --> 00:29:01,140 the back here 816 00:28:58,260 --> 00:29:04,620 microphone working its way up 817 00:29:01,140 --> 00:29:07,860 here we go uh so you mentioned uh 818 00:29:04,620 --> 00:29:10,919 regulation as one possible way through 819 00:29:07,860 --> 00:29:13,700 this what about organization and 820 00:29:10,919 --> 00:29:17,159 unionization 821 00:29:13,700 --> 00:29:18,840 that would be lovely I don't feel like I 822 00:29:17,159 --> 00:29:20,700 have a direct like there's very little 823 00:29:18,840 --> 00:29:22,020 unionization happening in Tech in North 824 00:29:20,700 --> 00:29:24,480 America which is where most of my 825 00:29:22,020 --> 00:29:26,760 working experience has been so I don't 826 00:29:24,480 --> 00:29:29,279 know what that would look like it would 827 00:29:26,760 --> 00:29:32,279 be a really really dramatic power change 828 00:29:29,279 --> 00:29:34,679 like across the board in terms of who is 829 00:29:32,279 --> 00:29:38,580 deciding how the company is spending 830 00:29:34,679 --> 00:29:40,260 human time and money uh but I would love 831 00:29:38,580 --> 00:29:41,820 to see that I don't I don't know what it 832 00:29:40,260 --> 00:29:44,460 would look like 833 00:29:41,820 --> 00:29:45,720 okay that is all the time we have thank 834 00:29:44,460 --> 00:29:47,340 you very much Courtney everyone please 835 00:29:45,720 --> 00:29:49,520 join me in thanking Courtney for your 836 00:29:47,340 --> 00:29:49,520 time