JCCC3 Tech Post-mortem (Twitt-arrrr, Cruise Monkey)

edited February 2013 in JoCo Cruise
While there were issues out of our control (Apple making using the wifi without Internet difficult) and ones with no ready solution (overloading the access points in the theater and game room), what did you encounter that could have worked better?

For a start I'd like PMs to work better in Twitt-arrrr, and perhaps a way to turn off threading. Any other suggestions?

«1

Comments

  • edited February 2013
    The annoying thing, for me, was that just the presence of a substantial number of monkeys (e.g. in the game room, the auditorium during concerts, or even the main dining room) crashed the ship's access points. When I got a text that required me to log on to fix a problem at home (fortunately, it only took a few minutes each time), I literally had to flee the room.

    I suspect that much of the congestion was Apple devices trying to use the exceedingly wasteful and chatty "Bonjour" protocol, even if their owners were not using the Internet. We might want to look into setting up our own, more robust routers -- preferably ones that block and kill Bonjour -- in areas where most of the communication is just with Twit-Arr and Cruise Monkey.
  • A dual-radio 802.11n access point would potentially make a big difference if it was possible to have a couple special ones set up in those spaces. (At least the game/conference room.) At least a good chunk of devices could flee to the 5GHz range in that case.

    I'd be surprised if simple mDNS activity was enough to overload an access point, but it would be an interesting experiment to find out. I wonder though if blocking it at the router would help, since at that point the device has already claimed radio time to attempt it. I don't know how much overall bandwidth usage would be dropped if it was just every device shouting alone into the wind.

    I've seen claims that a single 802.11b/g BSS starts to suffer some degradation beyond about 25-30 simultaneous clients but have only seen this referred to anecdotally. The collision avoidance algorithms and trying to give each device their turn, plus the big timeslices for non-client communication and client subscription can only do so much. (After examining these protocols at just a basic level I was left a little impressed that it ever works as well as it does. There's a lot of time spent just waiting, and neighboring interference just messes things up more.)

  • I do not have an iPhone.  Does anybody know if apps have enough access to make the settings changes necessary to override Apple's configuration difficulties (without needing a rooted phone)
  • I'm not an IOS programmer but I don't believe so. This was a thing Apple used to have a somewhat-buried option to work around but it disappeared in an IOS update and is apparently back in the latest iPad IOS fork? (I think, I hope I'm not misremembering what Ray told me about it.) Hopefully by the next cruise, Apple will have fully restored that missing feature to current IOS devices.

    The workaround our hard-working tech monkeys put in place this time worked perfectly fine for me and my iPhone.

  • I gave up on the Mustard app for Android because on the ship it seemed to load a page of posts and then stop.

    Given that, I second the call for the ability to turn off threading. What would be best is to be able to quickly and easily go back and forth between threaded and not threaded.
  • I really don't know what the situation is with iOS right now. Other people had the identical device and iOS rev and had the Auto Join switch, and I didn't.

    I would like to look into adding extra access points in the conference center and maybe on the stage. There are network jacks in the conference center so that should be easy, stage is unknown.
  • Oh is *that* what the Auto-Join switch was? I actually had that on my iPhone 5 but still followed the proxy directives. #SMRT  (Although I don't see it now, at home. Maybe only shows up for insecure networks?)

    For reference I have a Verizon iPhone 5. Could there be some slight IOS difference between the Verizon and AT&T models?

  • edited February 2013
    As a user of twittarr on iphone (AT&T, iphone 5, iOS 5.0.1 I think and always in airplane mode with wifi turned on), once I got the silly proxy working (goddammit, Apple, that is the stupidest wifi setup behavior ever), it mostly just worked. Occasionally it got confused and tried to log onto the "real" internet (popping up the login dialog), but refreshing a few seconds later fixed it. I did notice occasional cases where the wifi was clearly just broken but it was in big rooms of monkies (such as main concerts, though I didn't twirtarr that much in them so it wasn't ever annoying) so I assumed network congestion.
  • I'd love to see Twitt-arr integration in the CruiseMonkey app, rather than using Mustard on Android. Mustard is fine, but it doesn't have a threaded view, and the browser view is fine, but the default Android browser on my phone had trouble rendering the page while Chrome was extremely unstable on my phone. Aside from that, I thought the tech aspects worked flawlessly and I'm so happy that you guys put so much hard work into it all. I am in awe of all of this. Now the question is should I uninstall CruiseMonkey until next year or should I let it sit there, taunting me?
  • I hate to say it, but I think we've outgrown Statusnet.  Even when we weren't overloading the wifi by gathering in mass, it wasn't uncommon to see a post get buried in the public timeline in a matter of minutes.  I'd like to see something with better post organization, notifications, and private messages at the very least.  I have no idea what that would be or how viable it is, but that's what I would like to see.

    CruiseMonkey was mostly fantastic.  I only have some nitpicky suggestions: combined schedule viewer, a means of navigating the schedules by day, and some sort of icon to indicate "Hey, I'm not connected to the server and using offline data".
  • As someone who was a little responsible for Twitt-arr, I felt a little embarrassed at the state of things as I tried to use it.

    I think for next year we either need to roll our own Twitter clone. The code for StatusNet is so obtuse that I think trying to make it do what we want will be much harder than starting from scratch. In the next month or so I am gonna start working on something simple and build up based on whatever functionality we think we need. I am not saying we definitely need to use the thing I will build, but it will be there as an option depending on what everyone thinks.
  • I'd agree with the Statusnet pain - it seems like software that's optimized for tens of posts a day, not hundreds. It definitely worked, but I know there were things I missed at times. It was still fun and there was some great moments on it (cheese plate pimping itself was awesome) but I'd agree that it's unwieldy.

    @alice - if you want some collaboration I'd be happy to help - just throw it up on github or some such and I can sling some code. I know quite a few frameworks and can learn just about anything else.
  • Good grief, you have nothing to be embarrassed about! Improvements are always appreciated, but it impressed the hell out of me that it worked as well as it did, especially given the complications of deploying it and getting it all working on the ship.

    In terms of rolling our own, what's the status of identi.ca at this point? Wasn't it a fork of status.net of some kind?

  • edited February 2013
    @Thalandor46 Cruisemonkey for Android had an icon in the upper right that told when it was offline. As for combined schedules, you had the ability to favorite events and view them in a list. What I'd like for the Calendar is a grid view. A list view is nice, but it would be cool to see a true calendar view.

    EDIT: @aliceandstuff You have nothing to be embarrassed about. Having a Twitt-arr was better than having no Twitt-arr. You guys did an awesome job building a tool that helped all of us immensely. You have my eternal gratitude. :)
  • If that was the case with the Android client, then it wasn't functioning properly.  I had a green icon throughout the entirety of the week, but it wasn't until around Wednesday that I realized my settings were pointing to the old server.  Maybe that icon was just saying that my wifi signal was active?  I did not know about the favorites thing.  It certainly wasn't intuitive, and could probably be done better.  Again, nitpicky.

    And to be clear, I was thrilled to have Twitt-arrrr this year.  I would certainly be lost without it.
  • It we are going to roll our own Twit-Arr server and client, please share it out so that we can all work on it. I will gladly help to get an awesome experience.
  • As far as I know, identi.ca = StatusNet, and it is no longer maintained, in favour of pump.io.

    The problem with posts falling off the front page is a problem inherent to microblogging, not just StatusNet.  Like Twitter, if a lot is happening all at once, things fall out of view very quickly and it is up to the reader to be aware of this and read as far into the stream as they deem necessary.  Microblogging is not meant to be a way to ensure that all information can be tracked and read.  Web forums are a better format for that, where you can track read and unread threads.

    Some of the technical issues that I encountered were:
    * saw some HTML errors when trying to read the "G" section of the user directory
    * got some error dialog popups when trying to create an Event on my Android
    * PMs (obviously) are needed, or some other 1:1 / 1:few messaging (such as email/calendaring)
    * inconsistent Wi-Fi options on Apple devices (as mentioned by @origamislayer)

    I would love to have Exchange available for the general population.  Many of the features we would like to have for Twitt-arrr (notifications, PMs, reminders for events, etc) are all part of Exchange.  It would also help keep some of the stuff intended for a limited audience off of Twitt-arrr, and prevent it from pushing more general notices off the front page.

    A few times I noticed that Twitt-arrrr seemed quite slow and when I ssh'ed into server, I saw that the load was exceeding 1.0.  There are some inefficiencies that we can probably eliminate, such as
    * installing a PHP accelerator
    * configuring squid to not cache
    * serve up proper thumbnails
    Those three things alone would probably be sufficient performance improvements.

    One of the things that negatively affected most people though was the lack of detailed instructions available for people from the outset.  This was more of a logistical issue than a technical one.  On JCCC2, all required details were hammered out before hand so that documentation with the correct IP address could be printed out in time for onboard registration.  This year those details weren't set until we got on the ship.  The good news is that once we did, we got DNS and we got an extra static IP and then were able to include this info in the Sea Monkey newsletter (which despite the occasional typo or misprint, was a huge benefit).

    So while most post-mortems (especially technical ones) tend to focus on the negative, I would like to point out that I thought this year's Twitt-arrr was a resounding success.  Compared to last year, I saw far more people using it.  I had several excellent IRL interactions that couldn't have taken place without it.  Even if nothing changed for JCCC4, I think it is still a useful tool to make available.
  • @Grimoire, I agree with everything you've said.
  • I think twit-arr was great and worked better than last year. Last year it was a major struggle to get connected at all but this year it worked most of the time.

    Initially I was using mustard, but without threading it was impossible to keep up with the conversations. The technical problems I remember having with the browser version were : 1. It would throw an error when trying to RSVP to an event and 2. When I switched to desktop view to change settings then tried to go back to mobile view it seemed to get stuck in desktop view. That could be some issue on my phone, I didn't worry about it to much.

    If we could get it so the conversations were threaded but collapsed by default like on twitter then I think twit-arr would be close enough to perfect for me.
  • Everything I used, worked.  That was nice.  I was on wifi temperance brigade on previous cruises, but this time I wanted to make sure I was informed as to what was/had been going on.  I was satisfied that I could check once or twice a day and get a summary of events and news via CM and Twit-arrrr.

    My primary frustration with Mustard was that it didn't save my place.  With Plume, I can save my place and catch up on what I missed very easily.  

    The other frustration with Mustard was that it sometimes would not refresh any earlier posts, so I was stuck reading only the past X posts and couldn't go back further.

    Threading would have been nice in Mustard, as well.  The web version didn't want to load the couple times I tried it in Chrome on my Nexus, so I just stuck with Mustard.

    In CM, it would be nice for the calendar views to default to the current day.  I agree with an integrated official/unofficial calendar, too.  Also highlight what is going on right now/SOON.  An integrated communication client would be very nice, particularly if we could post links directly to calendar entries.  

    Would installing Vanilla/phpBB/VBulletin, etc be better/worse/different than creating twitter-like app from scratch?  I think it would be slightly heavier web traffic, but much more organized and easier to find posts about specific topics.  If there are clients for those, or a client could be rolled into CM, it would be similarly light.

    I would like to see more pre-loaded file hosting for things such as a copy of Wikipedia , a copy of IMDB , a copy of JoCoCruiseCrazy.com , e-copies of the Sea Monkey, price lists/locations for merch, board game manuals, puzzle references, cat videos, meme macros, etc.  Also, maybe an official announcements page for major updates/events/cautions/lost&found, etc.

    The other major tech request is a more efficient way to dump/share videos/pics.  Roop's 4TB drive was great, except that some of us didn't bring PCs, those that did found that transfer times for data were longer than expected, and it was mostly single access.  There should be a solution that addresses some of those issues.
  • edited February 2013
    "A few times I noticed that Twitt-arrrr seemed quite slow and when I ssh'ed into server, I saw that the load was exceeding 1.0. "

    Did you see where the load was primarily coming from?  System time, user time, I/O wait time, etc?  Also was that a system where the load average was a raw value, or one where it was divided by the number of logical CPUs?

    Was the laptop running off of spinning platters or SSD drives?
  • @mikesphar I was smart enough to install sar when I first got on the box. I'll go back and see if I can figure out where the problems were.
  • edited February 2013
    Everyone: The domain name "twit-arr.com" is still out there, but obviously not doing anything useful right now. LMK if you'd like me to add host names that point to the arrchive, a test server for future incarnations, etc.
  • edited February 2013

    @mikesphar The laptop was running from spinning platters, although the virtual machine running Twit-arr did have a dedicated drive. I could have given it another processor, but I didn't hear about the load issue until the last day.

    I also hope we can do Exchange for the entire Monkey population in the future. It would solve a lot of problems in terms of being able to do customized calendars and real 1:1 or 1:few communications. The client interaction is heavily optimized for mobile clients, so load on the wifi should be easily less than what we are seeing for Twit-Arrr.

     

  • I'm on Android and Twitt-arrr via the browser worked amazingly for me. I don't know enough about All The Things that had to happen in the background, and I don't have anything constructive to add, but I wanted to say THANK YOU to everyone involved for making it happen! You all put so much work into this, and it looked great and worked well. Considering the alternative (i.e. costly ship wifi), it was an astonishing achievement. Thank you.
  • Overall? SO much better than last year. Alice's front-end was easy on the eyes, and all the Mysterious Sparky Bits going on behind made it simple for this caveman mechanical engineer to connect and do what he needed to do.

    I'm intrigued by the idea of making our own code though I have zero idea how to do it -- but based on the talent pool present, I'm sure something ginned up by Alice and Team Twit-Arrrr would be amazing. :)
  • Was it possible to use the app to consume Twit-Arr? I never got it to work, if so.

    The other thing was that it took me most of the week to grok the difference between my "Home" link and the base link of the app (ie, feed of people I follow vs. public feed). Had I noted that earlier, it would've been even MORE useful.

    I'd still love to see a way for us to get proper email onboard, but I understand RCI's reluctance to allow it.
  • @chetman: Cruise Monkey wasn't meant to really be a twitt-arrr client, though it's been kicked around for the future. There is an official status.net app, but they broke it with a change in the server a year ago and have pretty much decided to just abandon it. In general I'd prefer a client to the web page, but getting one to work wasn't in our time frame.
  • Glad I wasn't just being stupid. ;)
  • I am not a Linux admin and I didn't think to look for what was the choke point.  Fortunately, @origamislayer is and has us covered.  :)
  • Indeed, the sar data should hopefully be informative if it's any one particular thing.

    Also given just the other day I saw an online special for a 256GB crucial SSD for about $150 after rebate...maybe we can pass the hat around for an SSD drive or two by next year.

  • edited February 2013
    The one thing I know I must bring next time: my Jambox.  The AV connector situation sucked, but with a jambox (or alternative ultra portable speaker) we could alleviate the problem greatly regardless of the room situation.

    Also, I do Android dev for a living so if we wanted to write our own client (or improve an existing one) I'd be totally game.  The browser was good enough but there are some cool things one could do with an actual client (like thread watching/notifications).

    (Note: I didn't try the actual Android app while I was onboard - I know, I'm lazy.)
  • I have iOS 6 on my iPhone and iPad. iPad had the auto join switch, iPhone didn't.

    Twitt Arr worked much better than last year. If we code our own, we should think about a functional spec first.

    What the status.net gives us seems to a flat forum in a twitter stream, so things scroll off quickly.

    Do we want a structured forum like this one? A simple twitter stream? Both?
  • edited February 2013
    I personally like combs and trees better than flat forums. In the early days of BBSes, I worked on code for several tree-structured boards, including LBBS and Stuart ][.
  • I had no issues with twit-arr on the ship  :)  I liked Mustard's notifications too.  I would rely on mustard to notify me, then twit-arr to see what was said.

    I think a forumish thing would possibly serve our needs better.  Maybe a forum with a shoutbox?  That would allow better threading and avoiding of getting topics that need groups of people to see it from getting lost, then a shoutbox would allow the people to just yell how awesome everything is  :) 
  • One thing stated early on in last year's planning was that we wanted organizing tools, but didn't want to give people too many excuses to stay in their cabins and play on their laptops. That's the one thing that makes me wonder if a full forum would be too far. Given the power of @gbasden's computer, we could probably pull off anything short of a media server (don't clog up the wi-fi with videos).

    I would like to un-thread Twitt-arrrr to make it more Twitter-like and easier to read. Notifications (and maybe a good client for everyone) would make me very happy as well.
  • Bah, I don't read twitter itself for the unthreaded reason  :) 

    I find it hard to believe though that people won't want to come out of their cabins anyway, even if there is a forum  :D 
  • For the kind of "seat of the pants" event-based stuff, I much prefer twitter style straight timeline than navigating a tree structure.  I feel like I'm spending all my time navigating around to figure out what's "new."  Twit-arrrr is better than no Twit-arrrr, but status.net seems to be the worst of both; it was maddening to see the same ShipPig poll over and over again as someone answered it when people were talking about interesting stuff 10 minutes ago but it's scrolled off to the 2nd page.  I like the way "real" twitter lets you see the conversation/context if you click on a message, but otherwise lets me start at "last thing I read" and scroll up until I'm caught up.

    I generally hate *hate* forums anyways; GamersWithJobs.com (and to a lesser extent, here), are the only ones I read because the content overcomes my dislike of the format.  ;)
  • Yeah, I agree. We don't need a "real" forum onboard. We just need MAYBE some shared event calendar added to TwitArr, and better dissemination of how to use TwitArr to best effect. 

    By Wednesday, once I figured out the distinction between the public timeline and "home", Twit-Arr was WAY more useful to me.  
  • Yes, a non-threaded timeline, ala actual Twitter, with the ability to trackback, would be great. Or at least options for each style.  Also, if users were alerted that they had DMs, that would be something (though a little difficult without email notifications maybe). I used DMs the first few days to try to organize with some people (@chetman included) only to find out that they didn't realize they had any messages.  I changed tactics and started putting things directly into the public timeline with @ callouts, but even then things were lost quickly in that stream.

    All of my whining aside, what we had, between the Cruise Monkey app and Twit-arrr, was an amazing set of tools for being in an environment that didn't have any of that available before we arrived or after we left.  Thank you and keep up the excellent work code and IT monkeys!  Hopefully there will be extra bananas in your rations next year.
  • edited March 2013
    In my poking around last weekend to see if there was a nice way to easily export a static copy of Twitt-arrr (there wasn't, really), I noticed a variable that seemed to be related to threaded vs non-threaded mode.  I will see if threading can be disabled via configuration.
  • @profbeard, I think part of our problem was that we didn't talk about how to use Twit-Arr -- I think you knew more about it than I did. 
  • I used Twit-arrr on two different Android phones (Gingerbread) and a Tablet (Ice Cream Sandwich). The only place I had trouble was the main theater. I never could get the web page to refresh there. I used only the browser and overall it worked well for me. 
  • My Motorola Droid (original - yes i'm cheap) could NOT stay connected to Twitt-arrr.  It would connect, load about half the page, then get tired and stop.  My laptop never had any issue even though they were right next to each other in the room.  I have no idea why the phone had such problems :(  I assume it's an age thing.

    Also, I was never able to get Cruise Monkey to load.  It would fail every time I attempted to open it.  Again, I assume it's the age of my phone/operating system that is the problem.

    Just a heads up to people.  I don't expect anyone to support ancient phones like mine.

  • @ducky: Was your laptop attached to the wi-fi or were you using the Ethernet jack in the cabin? It could certainly be the age and rev of Android on the phone, but I suspect a lot of our issues were overloaded access points. We have some thoughts about that for next year...
  • edited March 2013
    Actually, the DHCP pool was being exhausted anywhere there was a congregation of Sea Monkeys. Wired, wireless -- it didn't matter, because they were both hitting the same DHCP server for addresses. Most of the folks with smartphones had iPhones, which are brutal; they try to stay connected all the time and are constantly chattering.

    What's more, our cabin had neither a Wi-Fi signal nor a wired Ethernet jack.

    I found that my best option was to flee to an area where there was Wi-Fi but not too many monkeys. The Crown Viking Lounge was a good spot.
  • edited March 2013
    The DHCP pool should theoretically have been the same for the entire ship, no? So it shouldn't matter where people were congregated. (Almost by definition, an Extended Service Set area should allow any IP to roam to any access point. Otherwise you'd be resetting people's open TCP sessions as they roam around. I don't think you could accomplish that with different DHCP scopes on different APs.)

    I at least did not experience behavior typical of en exhausted DHCP scope (staying associated to the access point but getting a 169.* address) though it's certainly possible there were times it was happening and I was just lucky. (But not having enough DHCP addresses is seriously bush league networking. I would never blame clients for bad network design.) My outages seemed more typical of too many associated devices on a single AP. (Staying associated and keeping an IP but just a lot of timeouts/packet loss and general poor bandwidth.) Those mostly only happened to me periodically in the big theaters, especially right around intermissions or when a song ended, unsurprisingly. Most likely the times a lot of people were checking their phones and trying to update twitt-arr, I presumed. That's definitely a case where fleeing to a less populous area would likely help.

    Just for reference, Cisco says this about their Aeronet access points: Ideally, not more than 24 clients can associate with the AP because the throughput of the AP is reduced with each client that associates to the AP. They'll allow a lot more than that obviously, several hundred at least, but loss of performance starts pretty early. CSMA/CA is decent for avoiding collisions and the hidden node problem, but it's not ideal for several hundred devices vying for air time at once on a single channel. I recall stories of earlier Aeronet models actually crashing and rebooting themselves under the load of a couple hundred simultaneous clients.

    Getting several hundred devices onto the same wireless network on the same space with decent performance is actually pretty hard. I'm honestly not sure how to do it on 802.11b/g given the channel limitations, but it's fortunately something I've never had to do.

  • edited March 2013
    I've done WiFi for conferences, and -- yes -- exhaustion of an AP's maximum number of associated stations or its wireless bandwidth could have been an issue as well. However, at least in some cases I observed the same problems even when I plugged in.

    On HAL, there really was only one subnet and one DHCP pool for the entire ship, and so when the DHCP pool was exhausted I "borrowed" an unused IP outside the pool.

    This didn't work on RCI, because the subnets in different locations contained different addresses and the APs were set up as routers, not bridges. So, I couldn't set my Wi-Fi adapter to one address and leave it that way.

    When I moved to an area where there were fewer monkeys, I was able to get an address from the local pool and everything worked. Alas, this meant leaving the group when I had an emergency to attend to at home. I did my best to minimize the number of times I had to do this.
  • "This didn't work on RCI, because the subnets in different locations contained different addresses and the APs were set up as routers, not bridges."

    That is messed up. It means if you roamed the ship with open TCP sessions they'd get broken as you moved around! It sounds like they had the ship set up as a bunch of separate BSS's with the same SID, rather than as one ESS. God forbid two APs with overlapping areas were configured that way, you'd get active sessions disconnected any time the client decided the other AP's signal was better and re-associated.

    I don't know if that's in violation of any particular specifications, but it feels like it should be.

  • @mikesphar You're correct; you couldn't roam seamlessly from one AP to another. But then, that's not a big problem in most situations. And in some cases, it is advantageous. Some clients have poor signal strength detection and can "thrash" between APs. When we set up hotel Wi-Fi systems, we frequently do it that way.
Sign In or Register to comment.