Sep 16 11:19:26 hmm Sep 16 11:19:35 new ssh remote exploit rumors on full disclosure Sep 16 11:29:23 there is indeed one of those... Sep 16 11:29:32 * deviant- can't give details. Sep 16 11:29:51 deviant- looks like fbsd patched it Sep 16 11:29:56 Edit src/crypto/openssh/buffer.c Sep 16 11:29:56 Edit src/crypto/openssh/version.h Sep 16 11:30:12 so perhaps it's already become public Sep 16 11:30:16 snooze: how nice for them :P Sep 16 11:30:35 deviant- are the files that need to be patched? Sep 16 11:31:06 snooze: you lack a subject in your query. Sep 16 11:31:27 or, rather, a, uhm... Sep 16 11:31:28 are those 2 files that need to be patched to prevent this expoit Sep 16 11:31:36 what, like I can talk about it? Sep 16 11:33:39 Only version.h needs patching; buffer.c is a red herring. Sep 16 11:50:46 20hardknocks# tcpdump -i tun0 port 22 |grep -v earthquake.boredom.org Sep 16 11:50:53 let's see if anything bad happens Sep 16 11:52:09 did someone just kill sshd on boredom? Sep 16 11:52:16 not intentionally, *sigh*. Sep 16 11:52:31 getting hold of wankel now, hopefully. Sep 16 11:52:49 Maybe I'll put sshd under supervise tonight :) Sep 16 11:53:59 mendel seems like a good idea, think this is the 2nd or 3rd time someone goofed and killed all sshd processes, instead of just the main one Sep 16 11:54:28 heh.. that sucks. Sep 16 11:54:34 Well, I wish I knew /why/ that happens Sep 16 11:54:48 but yeah maybe I'll put webssh:443 under supervise or init or something :) Sep 16 11:55:10 111/tcp open sunrpc Sep 16 11:55:17 Well, should be easy to get in, right? :) Sep 16 11:55:32 mendel: yeah. Sep 16 11:55:53 yeah to the supervise suggestion, that is. Sep 16 11:55:56 What's port 813? Sep 16 11:55:58 although I don't know how much it'll help. Sep 16 11:56:20 deviant -- Quite a bit, really. We supervise an sshd on 2222 on our machines because it's *really hard* to accidentally kill off a supervised sshd. Sep 16 11:56:22 can you ftp in as root? Sep 16 11:56:31 and add a restart to the crontab? Sep 16 11:57:18 @home's noc had sshd run out of inetd, it made the initial connection take a few seconds longer Sep 16 11:57:48 Snooze: well they could do that for a sshd on a diffrent port Sep 16 11:57:55 a 'emergancy' sshd Sep 16 11:58:53 dsm teaching them ssh -p 2342 would have been too difficult Sep 16 11:58:55 amirgancie Sep 16 11:59:08 * Maestro licks jenn Sep 16 11:59:15 * jenn crawls onto maes Sep 16 11:59:24 * Snooze2 has his eyes pop out Sep 16 11:59:33 who is "fixing" ssh Sep 16 11:59:47 where? Sep 16 12:00:28 jenn on earthquake, deviant accidently killed off all sshd Sep 16 12:00:37 you mean the sshd Sep 16 12:01:05 unless multiple sshd run on earthquake Sep 16 12:01:10 Yes. Sep 16 12:01:15 but, can't fix it now Sep 16 12:01:17 main daemon, and all child processes Sep 16 12:01:17 ? Sep 16 12:02:00 you mean the "newly patched" sshd didn't start gracefully Sep 16 12:02:11 * Snooze2 shrugs Sep 16 12:02:13 or perhaps, didn't start at all Sep 16 12:02:14 idaknow Sep 16 12:02:32 * Snooze2 guess step 1 was "killall sshd" Sep 16 12:02:52 i've always had bad luck rebuilding/starting sshd remotely Sep 16 12:04:10 seems to have worked here Sep 16 12:04:12 root 30906 0.0 0.4 2592 2024 ?? Is 8:56AM 0:00.61 /usr/sbin/sshd Sep 16 12:04:12 hardknocks# kill -TERM 30906 Sep 16 12:04:12 hardknocks# /usr/sbin/sshd Sep 16 12:04:39 so now there's no root users logged in Sep 16 12:04:41 able to restart it. Sep 16 12:04:50 there's no-one logged in, because there's NO FUCKING SSH. Sep 16 12:04:59 joy. Sep 16 12:05:06 Get what you pay for. Sep 16 12:05:11 heh Sep 16 12:05:13 can't pkwong effect a reboot? Sep 16 12:05:37 ive never had problems with it. HUP session leader sshd and at least for recent versions of openssh it will restart. or just kill the session leader and restart sshd. did someone do a pkill sshd or somthing? Sep 16 12:05:44 dsmouse: Er, the machine is running. Sep 16 12:05:46 mendel isn't it the "router" again? Sep 16 12:05:49 mornin guys Sep 16 12:05:56 snooze, i bet there's a sshd parent that you could kill rather than that child Sep 16 12:05:58 The problem is not that we don't know what the problem is. Sep 16 12:06:06 The problem is that there is NO FUCKING SSH. Sep 16 12:06:08 * Snooze2 pats mendel on the head Sep 16 12:06:15 mendel: but a reboot will fsck, and then start the services Sep 16 12:06:20 blame it on the router, people will believe it Sep 16 12:06:25 Why would sshd start the second time if it didn't start the first time? Sep 16 12:06:40 While I would *appreciate* an indeterminate system right now I don't think we have one Sep 16 12:06:48 mendel++ Sep 16 12:07:06 mendel: I thougt it was killed, not that it didn't come up on a reboot Sep 16 12:07:14 It didn't come up on a restart. Sep 16 12:07:38 system restart, or `service sshd restart` Sep 16 12:07:46 * Snooze2 offers the seattle solution Sep 16 12:07:49 reboot it Sep 16 12:08:13 snooze: how's your classes goin? Sep 16 12:08:14 well, if the service dun start, you can keep rebooting it and waving a rubber chicken over it until it does start Sep 16 12:08:32 enjoy your week Sep 16 12:08:32 amy not bad Sep 16 12:08:38 I wonder what is on port 813. Sep 16 12:08:41 does the box by any chance at all have a serial console? Sep 16 12:08:49 even just a getty running on a serial port? Sep 16 12:09:03 perhaps you should set somthing like that up. Sep 16 12:09:16 Boy, everyone's a sysadmin. Sep 16 12:09:23 heh Sep 16 12:09:24 mendel: I will find out in just a few minutes, WRT 813 :P Sep 16 12:09:28 hee :) Sep 16 12:09:38 talon: serial to what? Sep 16 12:09:46 dsmouse: flood? :) Sep 16 12:09:53 but I don't think it has one. Sep 16 12:09:55 my turn is next Sep 16 12:09:58 dsmouse: doesnt matter even just a 1U box in the rack to ssh into and run minicom or tip or somthing. Sep 16 12:10:06 so you're not sure how you killed it, not sure how to fix it, wankel isn't anywhere, and it's basically dead. Sep 16 12:10:12 deviant-: so, would rebooting help? Sep 16 12:10:17 No, we're pretty sure how it was killed and how to fix it and where pkwong is. Sep 16 12:10:18 amy: you basically just got everything wrong. Sep 16 12:10:18 What's going on? Sep 16 12:10:25 zoratu: someone killed sshd Sep 16 12:10:26 And it's up and running for everything but ssh. Sep 16 12:10:27 <-- Snooze2 has kicked talon from #unix (shoot into the dark, see if you can hit the target on the other side of the building) Sep 16 12:10:28 So: No :) Sep 16 12:10:32 How did they do that? Sep 16 12:10:38 (If not deliberately?) Sep 16 12:10:43 deviant: then what is the correct version? Sep 16 12:10:52 zoratu: service sshd restart didn't restart Sep 16 12:10:55 --> talon (talon@fingers.shocking.com) has joined #unix Sep 16 12:10:59 See, the thing is, if we cover everything we're doing here, the backseat sysadmins get even more annoying. Sep 16 12:11:19 Who fucked it up in the first place? Sep 16 12:11:20 dsmouse: Ah. They didn't use reload? Sep 16 12:11:31 zoratu: dunno Sep 16 12:11:34 Nod. Sep 16 12:11:44 So did you hear about the new sshd exploit? Sep 16 12:11:47 No remote logging, I suppose. Or xinetd-driven rescue instance of sshd. Sep 16 12:11:59 i thought deviant claimed the title of "restarter" Sep 16 12:12:17 mendel: no Sep 16 12:12:23 zoratu: becarful, mendel gets pissed when you mention things like that Sep 16 12:12:28 Well then. :) Sep 16 12:12:46 mendel: i presume you mean something newer than last week? Sep 16 12:13:02 Today, AFAIk. Sep 16 12:13:14 though, it seems an unknown assailant was logged into earthquake, rebuilt a new sshd and attempted to replace the sshd with a patched build and failed miserably Sep 16 12:13:19 mendel: the one they just made a new release of openssh over? Sep 16 12:13:23 You think maybe? Sep 16 12:13:31 oh, that one. Sep 16 12:13:53 mendel you better listen to talon, he's a linux distro maintainer Sep 16 12:13:53 jenn: oh, the agony. Sep 16 12:14:05 heh Sep 16 12:14:08 * Maestro wonders why there wasn't a second sshd running during the sshd upgrade? Sep 16 12:14:19 maestro: There was. Sep 16 12:14:20 or mebbe that's just something I do Sep 16 12:14:21 It ain't now. Sep 16 12:14:22 I thought that didn't apply to the packages we were using. Sep 16 12:14:26 oy vey. Sep 16 12:14:27 amebosoft is like GMC, if they keep repeating that they're "Professional Grade" maybe people will believe it Sep 16 12:14:28 I guess maybe you just have NO FUCKING CLUE WHAT'S UP. Sep 16 12:14:32 zor: it most certainly did. Sep 16 12:14:49 Well, that sucks ass. Sep 16 12:15:19 * mendel moderates you all -1 Redundant. Sep 16 12:15:26 hehe Sep 16 12:15:27 amy, and we'll never know who this good samaritan was =) Sep 16 12:15:33 aww, someone's grouchy ;-) Sep 16 12:15:35 Hey guys, did you hear, there's a new openssh exploit? Sep 16 12:15:41 dave no way Sep 16 12:15:49 ahh. fuck.. Sep 16 12:15:50 dag, no shit? is it remote? Sep 16 12:15:51 but no worry! earthquake is secure! Sep 16 12:15:53 someone better rebuild the sshd on earthquake right away Sep 16 12:16:00 mendel++ Sep 16 12:16:00 Before it goes down! Sep 16 12:16:02 somebody set us up the sshd! Sep 16 12:16:11 ssh: connect to host earthquake.boredom.org port 22: Connection refused Sep 16 12:16:21 Well, ain't nobody gonna be haxoring earthquake's sshd. Sep 16 12:16:22 it appears someone found a local exploit on earthquake Sep 16 12:16:31 jenn++ Sep 16 12:16:32 * jenn whistles innocently Sep 16 12:16:33 jennr++ Sep 16 12:16:38 tsk tsk. Sep 16 12:16:56 * Maestro licks jennr again cuz she's tasty Sep 16 12:17:23 this hurricane preparedness is perhaps getting a little out of hand though Sep 16 12:17:30 sez you. Sep 16 12:17:42 earthquake's been down 2 days in a row =) Sep 16 12:18:05 jenn and still within earthquake's SLA Sep 16 12:18:10 earthquake was up yesterday Sep 16 12:18:17 you just couldn't get there! Sep 16 12:18:21 and it's up now too! Sep 16 12:18:23 It's up now! Sep 16 12:18:29 I can tell because it's giving me connection refused on port 22. Sep 16 12:18:30 snooze, indeed Sep 16 12:18:34 heh Sep 16 12:18:42 well if ya can't get to it via ssh, how about through telnet. Sep 16 12:18:48 * Snooze2 bonks amy Sep 16 12:18:51 Hahaha Sep 16 12:18:57 meanwhile, several dozen people all separately try to figure out how to remotely root a box they all care about Sep 16 12:19:00 amy++ Sep 16 12:19:00 nobody needs telnet because we've got sshd... Sep 16 12:19:08 * amy pats snooze on the head Sep 16 12:19:20 Oh hey. Sep 16 12:19:27 I bet someone has a mail filter, right? Sep 16 12:19:41 So just ftp in a new .procmailrc which fires off sshd when it receives a magic email message. Sep 16 12:20:07 * Snooze2 looks at dag Sep 16 12:20:10 good idea Sep 16 12:20:20 there's this whole business about "being root" though Sep 16 12:20:21 so, did whoever try to install a new ssh try to upgrade the RPM or just the binary? Sep 16 12:20:23 Snooze: I've done it before when I borked my home computer. Sep 16 12:20:33 mendel: sshd can listen on nonprivileged ports. Sep 16 12:20:44 dagbrown: Hard to login. Sep 16 12:20:58 dsm, that is the unknown. no one knows whoever Sep 16 12:21:17 mendel: Not so hard when you're only logging in as the user who's running the nonprivileged sshd. Sep 16 12:21:23 or are you using it in an "insider" implied way? Sep 16 12:21:25 I've done this before! I know it works! Sep 16 12:21:44 dagbrown: you have a account, go for it Sep 16 12:21:49 dave except that non-root users can't read the ssh*key Sep 16 12:21:58 oh, does ftp work? Sep 16 12:22:02 ftp works! Sep 16 12:22:04 manek@hardknocks:512:~>/usr/sbin/sshd -p 3000 Sep 16 12:22:04 Could not load host key: /etc/ssh/ssh_host_key Sep 16 12:22:04 Could not load host key: /etc/ssh/ssh_host_dsa_key Sep 16 12:22:17 * Dr-Radium casts detect remote exploit in progress Sep 16 12:22:19 Snooze: Tell it to read other ones that you also ftp in. Sep 16 12:22:24 ya right Sep 16 12:22:25 Snooze: use a diffrent config file Sep 16 12:22:30 heh Sep 16 12:22:56 so how many procmail files will we have launching sshd on 31337? Sep 16 12:23:03 Hahaha Sep 16 12:23:11 hehe Sep 16 12:23:39 so, doesn't flood nfs mount it? Sep 16 12:24:32 No. Sep 16 12:25:10 how about the hourly cron job that notices sshd not running and spawns off an emergency sshd? Sep 16 12:25:35 That'd be great if we had access to the system to install such a thing. Sep 16 12:25:43 dagbrown: ftp!1 Sep 16 12:25:55 i know Sep 16 12:26:18 why don't we just take a delorian, make it go 88 mph travel back 30 minutes Sep 16 12:26:43 * jenn notes from now on to call snooze "marty" Sep 16 12:26:52 bad mental image. Sep 16 12:26:57 snooze: and then kill the router so no one can update the files! Sep 16 12:28:38 * Dr-Radium casts detect procmail tunneling Sep 16 12:29:04 Oops, the ftpd on earthquake is configured to not let me in. Sep 16 12:29:20 dave: you sound so crushed. Sep 16 12:29:28 My life is over. Sep 16 12:29:33 dagbrown: I think that's why I asked the question earlier Sep 16 12:29:52 hmm Sep 16 12:30:02 immediate reject on me too Sep 16 12:30:07 (bad shell?) Sep 16 12:30:27 oh you're a bash user? Sep 16 12:30:56 wait.. deviant broke ssh? yet I managed to ID the right process and upgrade SSH on a lowly SOLARIS box just fine Sep 16 12:31:16 hmm Sep 16 12:31:17 did you take a picture of the process? Sep 16 12:31:28 is mankind not publically reachable? Sep 16 12:31:33 snooze: and I displayed it on my TELEVISION! Sep 16 12:31:40 mankind NO LONGER EXISTS Sep 16 12:31:40 way to go Sep 16 12:31:49 quick, *.boredom.org is down again, someone call Maxtor's RMA swat team Sep 16 12:31:49 :( Sep 16 12:32:49 this really sucks Sep 16 12:33:03 --> cst_ne (s@flood.boredom.org) has joined #unix Sep 16 12:33:14 uhoh Sep 16 12:33:23 guh, I have to go for a meeting Sep 16 12:33:30 <-- cst_ne has quit (Read error: 0 (Success)) Sep 16 12:33:51 So lemme guess, deviant said "rpm -e openssh" and then wondered why it said "connection closed by remote host", right? Sep 16 12:34:05 no one knows Sep 16 12:34:12 no one's admitting. Sep 16 12:34:30 dagbrown i suspect he's not going to 'fess up Sep 16 12:34:39 --> cstoneguh (s@flood.boredom.org) has joined #unix Sep 16 12:34:39 if it isn't a root user, then it's gotta be a local exploit Sep 16 12:34:49 guh so Sep 16 12:34:50 someone contact RH, have his RHCE revoked Sep 16 12:34:54 who broke what Sep 16 12:35:00 DEMOTED TO FIRST LEVEL TECH SUPPORT Sep 16 12:35:00 "someone" Sep 16 12:35:15 local sshd exploit found on earthquake Sep 16 12:35:17 cstoneguh deviant broked sshd when attempting to restart a patched version Sep 16 12:35:35 well, deviant hasn't said it was him exactly Sep 16 12:35:36 ? do you mean someone exploited it, or someone tried to patch and broke? Sep 16 12:35:37 NO NO Sep 16 12:35:37 was the old one accidentally killed or something Sep 16 12:35:43 "A mysterious hacker fucked up patching openssh" Sep 16 12:35:50 dagbrown++ Sep 16 12:36:03 * amy calls the enquirer Sep 16 12:36:06 they'll know. Sep 16 12:36:11 the exploit shouldn't crash the main server Sep 16 12:36:19 im guessing we will be hearing about this all day long. Sep 16 12:36:19 "a mysterious do-gooder tried to patch openssh on earthquake and fucked up" Sep 16 12:36:40 did they kill the old existing connections? Sep 16 12:36:43 who was that masked man? Sep 16 12:36:48 i prefer "enacted a local sshd exploit" myself Sep 16 12:36:49 cstone: Looks like it! Sep 16 12:36:54 euthanised their own connection before they could initiate startup of a new daemon Sep 16 12:36:55 arg Sep 16 12:37:24 08:52 did someone just kill sshd on boredom? Sep 16 12:37:32 08:52 not intentionally, *sigh*. Sep 16 12:37:32 08:52 getting hold of wankel now, hopefully. Sep 16 12:37:43 * dagbrown horselaughs Sep 16 12:37:46 that isn't admitting to it Sep 16 12:37:54 though it does seem like he has inside info Sep 16 12:38:24 * Snooze2 imagines chong saying "I was there man!" Sep 16 12:38:32 chong ain't here man Sep 16 12:38:41 Chong's in jail, man. Sep 16 12:39:41 host freechong.com Sep 16 12:39:41 freechong.com has address 216.136.224.156 Sep 16 12:39:53 workaround: don't send a programmer to do a sysadmin's job. Sep 16 12:40:17 amy's started in on the documentation Sep 16 12:40:19 fabulous! Sep 16 12:40:35 jenn: peanut gallery strikes again, y'know Sep 16 12:40:41 amy, of course =) Sep 16 12:40:54 * dsmouse throws peanuts at amy Sep 16 12:41:01 pistachios? Sep 16 12:41:03 yum. Sep 16 12:41:15 no, peanuts Sep 16 12:41:30 same difference. Sep 16 12:41:31 I throw pistachios at the pistachio gallery Sep 16 12:41:40 pfft Sep 16 12:49:38 I guess I should upgrade my opensshd, eh? Sep 16 12:49:44 --> ducky (chewie@adsl-67-64-91-13.dsl.austtx.swbell.net) has joined #unix Sep 16 12:49:48 dagbrown don't risk it, man Sep 16 12:50:03 dave: ohno's was done this mornin, so at least ya don't need to worry bout that side Sep 16 12:50:06 yeah, you wont get away with murdering him dagb Sep 16 12:50:06 i patched mine last night& it worked ok Sep 16 12:50:32 Patched mine at home (I think), building the one at work now. Sep 16 12:50:46 * talon is already working on it starting with the test boxes under his desk. Sep 16 12:50:54 Snooze: But it's so HANDY! Sep 16 13:56:29 --> wankel- (ident@ohno.mrbill.net) has joined #unix Sep 16 13:56:38 aight. who broke the box? Sep 16 13:56:51 deviant Sep 16 13:57:05 what'd he do, a killall sshd? :) Sep 16 13:57:07 --> homebrew (ethanr@ethanr.com) has joined #unix Sep 16 13:57:10 he won't say Sep 16 13:57:14 thbbt Sep 16 13:57:14 but i bet so Sep 16 13:57:46 he better not give me any more grief about how long it took me to upgrade the box a few times :) Sep 16 13:58:01 hee Sep 16 13:58:07 >wankel-< sshd upgrade borked Sep 16 13:58:07 wonder if he hit the same bug in the sshd init.d script that mendel and i told him about months ago Sep 16 13:58:10 --- dagbrown gives channel operator status to wankel- Sep 16 13:58:18 >wankel-< the problem is that 'sshd' seems to not run at all whatsoever Sep 16 13:58:19 the one he SWORE didn't exist Sep 16 13:58:24 wankel: No. Read your msgs already. Sep 16 13:58:30 sorry, lagged Sep 16 13:58:35 you just sent them :) Sep 16 13:58:44 well, it obviously doesn't exist as there is no sshd now is there? Sep 16 13:58:44 <-- siege has quit (Read error: 104 (Connection reset by peer)) Sep 16 13:59:07 earthquake is GUARANTEED secure from ssh attacks now! Sep 16 13:59:09 --> siege (siege@66.93.240.123) has joined #unix Sep 16 13:59:09 it if existed, there would be some evidence of an sshd Sep 16 13:59:15 <-- siege has quit (Client Exiting) Sep 16 14:04:37 <-- ducky has quit () Sep 16 14:06:35 is it back yet? Sep 16 14:06:46 is it back yet? Sep 16 14:07:15 Snooze: sit down and shut up or - SO HELP ME - I'll turn this net right around! Sep 16 14:13:03 hmm. we need to run some other sshd server on some other secret port or something :) Sep 16 14:13:09 we do! Sep 16 14:13:10 it got killed too Sep 16 14:13:13 or buy a console box. Sep 16 14:13:13 (webssh:443) Sep 16 14:13:20 or run telnetd Sep 16 14:13:22 no, a DIFFERENT server Sep 16 14:13:22 or run a cable to flood Sep 16 14:13:25 --- You have been invited to #boredom by deviant- Sep 16 14:13:33 <-- wankel- (ident@ohno.mrbill.net) has left #unix Sep 16 14:13:34 this is not new :) Sep 16 14:14:05 earthquakebot, run service sshd start Sep 16 14:14:14 give sigbus sudo Sep 16 14:14:23 sigbus restart sshd Sep 16 14:14:23 Snooze2: i'm not following you... Sep 16 14:14:35 it's on the wrong host anyways Sep 16 14:14:39 --> wankel- (ident@ohno.mrbill.net) has joined #unix Sep 16 14:22:59 --> Mennonite (Mennonite@x14d111.malone.edu) has joined #unix Sep 16 14:23:14 is earthquake's sshd down Sep 16 14:23:17 yes Sep 16 14:23:22 hrm Sep 16 14:24:23 no webmail anymore too :) Sep 16 14:24:34 i should probably get imp working again some time. Sep 16 14:24:48 imap should be up, though. Sep 16 14:24:51 imaps, that is. Sep 16 14:25:56 go fbsd! Sep 16 14:25:58 # kill `cat /var/run/sshd.pid` Sep 16 14:25:58 # (. /etc/rc.conf && ${sshd_program:-/usr/bin/sshd} ${sshd_flags}) Sep 16 14:26:07 that's from their advisory Sep 16 14:26:14 except that sshd is in sbin, not bin