19:00:34 #startmeeting 19:00:34 Let the Jenkins meeting commence! 19:00:56 #topic Moving cucumber from Contegix to the OSUOSL data centers 19:01:07 Scary 19:01:08 matrix ... will do thanks. 19:01:15 heh 19:01:17 awolf2: that's a plugin a take it? 19:01:30 softservant: mind holding off for a bit, we're starting up a project meeting 19:01:36 kohsuke: so all these agenda items are related 19:01:43 Yeah 19:02:07 I wanted to bring this up, because I'd like to consolidate our non-mirror infrastructure into the OSUOSL 19:02:16 well, non-mirror and non-builder 19:02:44 So in the sorted order, how do you see this happen in what steps? 19:03:00 Do we move off services from cucumber before it moves? 19:03:10 Or do we just take a hit for a few days (?) while the machine moves? 19:03:36 Or when we say "move cucumber", are you thinking about moving its functionalities but not the physical machine itself? 19:04:24 both 19:04:32 I would move services off, and then move the server itself 19:04:48 OK 19:05:16 the primary services are: www, mirrorbrain, jenkins master 19:05:28 there's a few other little things though 19:05:31 right 19:05:48 l10n submissions, log store 19:06:17 I'd be more than happy to run some things in AWS for the time of transit and migration 19:06:20 for the littler things 19:06:22 that's easy peasy 19:06:46 Sorry, I've been a bit out of the loop. What's the context for this move? 19:06:47 so, with this agenda item, I had no timeline in mind, I just wanted to discuss it with you, hare_brain and abayer 19:06:48 So let's think about big boys first 19:07:01 hare_brain: consolidation of infrastructure to the OSUOSL 19:07:13 and potentially decommissioning the physical machine cucumber 19:07:15 I meant, what's the motivation? 19:07:52 hare_brain: that's a good question --- status quo is the easiest 19:08:02 to consolidate things to the OSUOSL, cheaper, easier to manage 19:08:15 the status quo is crap 19:08:29 that machine runs pretty warm, and is already 3 years old or something like that 19:08:46 it can do things for a long time, but we ask quite a lot of it right now 19:08:50 When I asked that to rtyler, whether the motivation is the cost or the principle, I think he said both 19:09:23 The cost problem we should fix by having the project pay, if that hasn't happened already 19:09:48 And I understood the "principle" part to be "let's make sure we are working toward the right direction" 19:09:54 So crap could mean that you're not happy with the setup, but has it been causing problems for people? And by people, I mean committers and end users. 19:10:34 hare_brain: I do worry about a catastrophic loss of the server that will happen one day 19:10:45 it has not been causing problems for end-users or committers 19:11:03 It's a hodge podge of services that I don't think we know how to fully recreate if it goes to the server heave 19:11:09 n 19:11:11 * rtyler nods 19:11:17 that's exactly why I dislike it 19:11:17 :P 19:11:47 OK, so this is clean up to allow for improved "business continuity" for lack of a better term. 19:12:11 heh 19:12:11 That's my take of it 19:12:12 yes 19:12:48 so in terms of infra tasks 19:12:56 So what's the cost now, and what do you expect the cost to be on OSUOSL? i.e., what makes the new thing cheaper? 19:13:00 we need to move confluence off eggplant to a new VM 19:13:09 hare_brain: we don't pay for any services at the OSUOSL 19:13:15 we just beg people to donate to them :P 19:13:18 they're so nice 19:13:23 #info the OSUOSL is nice 19:13:37 let me pull up my trello board 19:13:37 And in terms of what we pay to Contegix, how much? 19:13:43 $60/month 19:14:04 Right, so money-wise the project can keep it as long as we need. 19:14:04 #info Jenkins Infra trello board 19:14:08 Is that coming out of your pocket? The project should reimburse you for that if it has been. 19:14:22 it has been, but I don't care enough about it to change it right now 19:14:42 anyways, what I'd like to do, is get the okay here to start migrating non-puppetized services off cucumber 19:15:05 with the long term goal of moving more services into "better long term" pieces of infra 19:15:23 That's nice of you, but a part of it is also about preventing the server from disappear when your plane crashes & burns 19:15:33 heh 19:16:14 So separate from the migration discussion, I think we want the server to be owned by the project, pay the cost, and etc. 19:16:46 * rtyler nods 19:16:49 +1 19:17:28 (And that reminds me of the domain name --- is that actually transferred to SPI, or is it still owned by you?) 19:17:57 I believe it's under my namecheap account 19:18:05 (every time I get SSL it has to go through your approval, so there's that bus factor) 19:18:12 * rtyler nods 19:18:21 kohsuke: neither of us are good at preventing jenkins bus factor 19:18:28 you can never come flying with me 19:18:29 #action kohsuke to ask SPI how to transfer domain ownership to SPI 19:18:31 :-) 19:18:37 good item 19:18:58 Now, on migration ... 19:19:00 #action kohsuke to ask SPI about best way to "own" hardware at Contegix datacenter 19:19:04 Yeah 19:19:09 I'm sure Debian is already doing it 19:19:30 Probably just have a spreadsheet somewhere that lists project hardware. 19:19:51 I need to find out how to set things up so that bills go SPI 19:19:57 * rtyler nods 19:20:17 hare_brain: do you have a trello username? I'll add you to this infra board 19:20:27 I don't think I do. 19:20:33 ok, we can resolve that later then 19:20:41 #topic Migration of services off cucumber 19:20:55 Or at least so that Contegix recognize SPI as the lawful owner (and maybe put me or somebody as the person who handles payment) 19:21:14 right 19:21:26 Right, on migration, can we use docker? I'm a fan boy now 19:21:39 heh 19:21:47 Makes adding new services & testing it easier 19:21:48 DOCKER IS NOT FOR PRODUCTION USE 19:21:53 it says it so clearly guise 19:22:15 the how I'm less concerned with 19:22:21 and that doesn't need to be discussed here 19:22:26 I think the what and when is more prudent to discuss 19:22:30 OK 19:23:03 in terms of things that aren't puppetized, I think i18n isn't, nor is robobutler 19:23:07 so I could start with those? 19:23:21 Sure 19:24:25 alright 19:24:29 To me, a big part of infra as code is to be mindful of where states are 19:24:43 #action rtyler to start puppetizing the i10n app to prep for migration 19:24:43 Just puppetizing it doesn't capture that. 19:25:00 that's more the ops part of devops ;) 19:25:38 the puppetization being the first part solves for the "continuity " problem even if we don't move anything off cucumber 19:26:02 anyways, I'll start on that, we can discuss on the infra@ list the mechanics of moving things at a later point 19:26:06 can we talk confluence now? 19:26:38 I'd feel great if we get to the point where state is backed up separately for each service, non-state for each service is Dockerized, and, we'd do a migration of a service from one box to another 19:26:46 Yes, OK, confluence 19:26:56 #topic Migrating Confluence off Eggplant 19:27:09 so, I "found" the machine again that the OSUOSL gave us solely for confluence 19:27:15 Yay 19:27:17 this is what happens when we don't write things down :P 19:27:22 Can you add my key while you still remember it? 19:27:28 Admit it, you never lost it. You were just using it to mine bitcoins. 19:27:28 yours was there already 19:27:30 but under the root user 19:27:36 which is the part I forgot 19:28:01 nope 19:28:07 ssh root@jenkins-confluence.osuosl.org refuses me 19:28:31 so I'd like to set up confluence on the machine based on our Puppet bits, and start pointing it the same DB 19:28:54 I'm not sure if that's a sensible idea, or wil confluence totally fuck our DB that way? 19:28:57 Err, I don't think Confluence would be happy to have two instances pointing to the same DB 19:29:08 crap 19:29:15 What's the DB? 19:29:21 Let me verify 19:29:30 some hosted OSUOSL hosted postgres DB IIRC 19:29:39 that's the other nice thing about consolidation to the OSUOSL 19:29:43 is they'll run DB servers for us 19:29:44 \o/ 19:30:01 Can you set up a replicated slave? 19:30:13 we probably do have access to such a beast 19:30:16 https://confluence.atlassian.com/display/DOC/Cluster+Checklist#ClusterChecklist-DatabaseSetup 19:30:29 So looks like I'm wrong 19:30:37 #action kohsuke and rtyler to work out plan to validate new Confluence instance in parallel with the existing one 19:30:38 But we do need to store attachments in DB, which we don't 19:30:52 I'll add a task in Trello about attachments 19:31:23 I still think it's easier to start by cloning a DB and then cut over later 19:31:33 #info We're planning on moving Confluence because the existing host runs Jira and Confluence and can't handle the load 19:31:39 not sure if replicated slave would work because new confluence will want to write, too 19:31:46 kohsuke: okay, we'll do that 19:31:55 kohsuke: do you have an expert at Cloudbees that could work with me? 19:31:59 your time is too sporadic 19:32:02 There's also the related static cache layer (and nginx) to go with it 19:32:26 We are hiring! Lookins gor a great ops guy to work with us! 19:32:32 Looking for... 19:32:39 heh 19:32:54 okay 19:33:00 The flip side is that we don't really have a full-time ops guy at hand 19:33:03 #action rtyler to find some sucker that isn't kohsuke to help with the confluence migration 19:33:09 But let me check with our acting ops guy 19:33:17 I think that covers my questions about confluence 19:33:25 last topic? 19:33:31 (I was suggesting replicated to set up new confluence in read only mode first, and then when we're confident of the new setup, flip the switch to make it the new writable master, and turn off writes on the old one.) 19:33:44 But if we just go with clustering, that is a better way to go 19:34:03 might be a good opportunity to also finally bite the bullet and upgrade to Confluence 4 19:34:09 * rtyler nods 19:34:14 I'm sure the devs will LOVE 4 19:34:16 * rtyler gags 19:34:26 The totally backward incompatible beast that still has the same name. 19:34:39 #topic more stickers 19:34:48 this wasn't on the agenda! 19:34:52 #chair rtyler hare_brain 19:34:52 Current chairs: hare_brain kohsuke rtyler 19:34:53 Yes it was 19:34:53 ORDER IN THE COURT! ORDER IN THE COURT! 19:35:02 oh 19:35:05 we skipped one 19:35:11 #topic Switching from masterless Puppet to a Puppet master. 19:35:17 yisss 19:35:20 Hahah 19:35:22 okay, this is more of an FYI 19:35:33 I've asked the OSUOSL for a new VM to run a puppetmaster 19:35:51 masterless puppet has served us well enough, but we're missing a LOT of inventory and status information from machines 19:36:01 like, did the puppet run actually work 19:36:33 I'm planning in the next couple months on moving more things under a puppetmaster, which would change our workflow from git commit + poll-based git pull deployments 19:36:51 to git commit -> testing in JonJ -> deployment of puppet to master -> machines updating 19:37:04 OK 19:37:10 so we'll get more inventory and status info, but also have a better delivery and quality pipeline 19:37:13 What would "test in JonJ" look like? 19:37:33 at the minimum, rspec-puppet, and catalog compilation 19:37:39 best case scenario, actual integration testing 19:37:43 TBD though 19:37:46 OK 19:38:13 now we can talk stickers :) 19:38:16 hold on 19:38:27 #action rtyler to keep the board in the loop on the puppetmaster migration status 19:38:50 #action rtyler reaching out to Puppet Labs to see if they will help set up a beautiful/ideal "Open source using Puppet" infrastructure with us 19:38:58 #topic More stickers 19:39:04 Right 19:39:15 So between FOSDEM and SCALE I've run out of all the stickers 19:39:24 except one or two packs that I still have at hand 19:39:29 So it's time to reorder 19:39:42 Let me pull up the price here... 19:40:03 2000 stickers for about $600 19:40:19 Any objection in spending that money? 19:40:54 I'm assuming it's OK 19:41:05 +1 19:41:15 can we get more for cheaper per-piece? 19:41:25 how long did it take us to burn through the last 2k stickers? 19:41:26 Just wanted to get some confirmation so that I won't be acting alone 19:41:40 The last order was Feb 2013 19:41:51 so 2000 stickers a year? 19:41:55 Yeah 19:42:02 seems reasonable, BUY IT NAO 19:42:20 try to get them to give you a special open source price or something thoguh 19:42:22 Tempted to ask what the run rate was at each event, but I doubt you were keeping track. :) 19:42:39 FOSDEM and SCALE are really the ones stickers fly away 19:42:44 hare_brain: FOSDEM we definitely burned about 500 19:42:54 I think in all the past years we've spent all that we brought there 19:43:00 * rtyler nods 19:43:03 So it's mainly constrained by how many we can bring 19:43:28 Other local meet-ups and JUCs, # of attendees are small enough 19:43:45 2000 has 85% discount and 3000 has 87% discount 19:43:56 So going beyond 2000 isn't really saving much. 19:44:12 * rtyler nods 19:44:23 Actually, I should check with that guy who offered us free stickers 19:44:43 He insisted that we talk this in e-mail and not in IRC, and sure enough I lost track of that conversation 19:44:48 when do we talk about cloudbees giving us some of their newfound venture monies? :) 19:45:08 high rollas 19:45:16 rtyler: it's coming: https://wiki.jenkins-ci.org/pages/viewpage.action?pageId=71435396 19:45:31 excellent 19:45:52 does that conclude the meeting? 19:46:11 I'll get stickers ordered in parallel to follow up with the other sticker guy 19:46:20 so that I have some supplies at hand 19:46:25 And yes, I think that concludes it 19:46:28 #endmeeting